AMORTIZED BAYESIAN META-LEARNING
Meta-learning, or learning-to-learn, has proven to be a successful strategy in attacking problems in supervised learning and reinforcement learning that involve small amounts of data. State-of-the-art solutions involve learning an initialization and/or optimization algorithm using a set of training episodes so that the metalearner can generalize to an evaluation episode quickly. These methods perform well but often lack good quantification of uncertainty, which can be vital to realworld applications when data is lacking. We propose a meta-learning method which efficiently amortizes hierarchical variational inference across tasks, learning a prior distribution over neural network weights so that a few steps of Bayes by Backprop will produce a good task-specific approximate posterior. We show that our method produces good uncertainty estimates on contextual bandit and few-shot learning benchmarks.

This work proposes an adaptation to MAML-type models that accounts for posterior uncertainty in task specific latent variables. This is achieved via a hierarchical Bayesian view of MAML, employing variational inference for the task-specific parameters. The key intuition of this paper is that one can perform fast and efficient test-time variational inference for the task-specific latent variables by learning a good initialization during meta-training. This is achieved in a very similar fashion to MAML, and allows for an interesting form of amortization of test-time inference.
Pros:
- For the most part, the approach presented is principled and well justified.
- The motivation is clear: in the few-shot learning regime we expect to have little data to infer the task-specific latent variables, and so we should perform posterior inference to account for uncertainty.
- The paper is well written, clear, and easy to follow.
Cons (more details below):
- It is not clear what the significant contributions of this paper are, as a number of methods have been proposed to account for uncertainty in the task-specific latent variables, and results for many of these methods appear to be better than those presented here.
- Experimental section does compare to many of the existing related methods
- There are some conceptual issues that need to be addressed by the authors.
I enjoyed reading this paper, and I think the ideas and work presented are, for the most part, solid. However, I am not sure to what extent the novel contribution of this paper is significant. Several papers, including Grant et al. (2018), but going back to Heskes (2000), have proposed the hierarchical Bayesian view of meta-learning. Grant et al. (2018) used a Laplace approximation to learn in such a model with MAML-type settings, presenting a method that accounts for uncertainty in this family of models. More recently, Finn et al. (2018) and Kim et al. (2018) have done this in a variational manner, albeit with variations in the implementation details. Gordon et al. (2018) proposed a more general presentation, unifying the above works (and others) in a Bayesian framework that allows for different functional forms of posterior inference (both point estimates and distributional) of the task-specific parameters, including gradient based procedures. All of these papers have been publicly available for a few months at the time of submission, such that this view of meta-learning as (amortized) Bayesian inference is not novel.
Here are some points that I would ask the authors to address during the rebuttal period:
- The method presented in the paper does not account for the meta-training splits into query and test sets, other than to mention that these led to empirical performance gains (this is somewhat typical of probabilistic meta-learning papers). However, it not clear that this is justified from a probabilistic inference perspective, which would favour conditioning on all available data at inference time. Further, in the experimental section, the authors state that "For the few-shot learning experiments, we found it necessary to downweight the inner KL term for better performance in our model". Put together, it is not quite clear exactly what form of approximate inference is being conducted here. Can the authors comment on this?
- I am not sure I agree with the authors' interpretation of the term "amortized Bayesian inference", at least in that it deviates from the way the term is typically used in the related literature. The method negates the need to maintain variational parameters for each latent variable, and approximate posterior inference for unseen tasks may be performed relatively efficiently, which is highly desirable. However, a gradient optimization procedure must still be performed for inference of task-specific variables for new tasks at test time. Thus, new variational parameters must be introduced and optimized at test time. It is true that by finding good global initializations the authors may drastically reduce the computational cost of the inference process, but this implies that the cost of inference at test time has been reduced, not fully amortized to a fixed cost (unless one fixes the number of gradient steps, which is a further deviation from variational inference and requires a prior of the form used in Grant et al. (2018)). Full amortization of inference for the task-specific variables is proposed by Garnelo et al. (2018) and Gordon et al. (2018), as well as Edwards and Storkey (2016), all of which employ inference networks mapping directly from the query sets to the variational parameters of the latent variables. In these cases, posterior inference of the latent variables for unseen tasks has the constant cost of a pass through an inference network, rather than several forward-backward passes, and does not require introducing new variational parameters to be optimized. Further, these methods negate the need for differentiating through gradient-based procedures at meta-training time, which is not avoided in this paper, but rather dealt with in the standard Hessian-vector product form. It would be highly useful in the paper (perhaps in the related work section) for the authors to conduct a more thorough comparison of their proposed method and the existing literature employing amortized inference for meta-learning, to put their work in context.
I also have a number of concerns regarding the experimental section of the paper, which I find to be lacking both in details and the empirical comparison of the method to existing works.
- The authors' cite recent works on meta-learning that take into account uncertainty in the local latent variables (e.g., Grant et al. (2018), Finn et al. (2018), Kim et al. (2018)), but do not compare to these methods.
- Results from Garnelo et al. (2018) are not provided for the contextual bandits experiment. Their results seem to be comparable or better to those presented in this paper. Can the authors comment on this?
- The same is true for the few-shot learning case, where MAML is the only method compared to, despite there being, at the time of submission, many papers which have significantly improved upon these results.
- In terms of details, it is unclear how many gradient steps were taken at test time, and how this affects performance of the model.
- In terms of accuracy, the proposed method appears to be under-performing significantly (i.e., below confidence bounds in almost all cases).
- The statement "...we believe improvements could be made with better variance reduction methods for stochastic gradients" should, in my opinion, either be investigated or omitted from the paper.
- In terms of uncertainty quantification, I find this experimental evaluation highly interesting. However, there is not a comparison to much of the existing work. The comparison to MAML is only of moderate interest in this case, as MAML is a deterministic method and is not expected to perform well in this regard. A comparison to Probabilistic or Bayesian MAML (at the least) would be more convincing if uncertainty calibration proved to be better for this method.
Overall, the paper proposes a principled approach to performing approximate posterior inference for task specific latent variables in meta-learning settings. The paper is well-written, and the method is clearly derived. However, it is my impression that the paper does not make significant novel contributions to the existing research in (probabilistic) meta-learning, does not properly acknowledge all existing work (much of which covers the main ideas presented in the paper), has a number of conceptual issues that might need addressing, and its experimental section lacks evaluation and comparisons to the existing similar works. As the method is, for the most part, principled and well-derived, and the paper well written, I am willing to reconsider my overall score if the authors can demonstrate either (i) significant novelty or (ii) that this particular flavour of inference for the task-specific parameters provides significant benefits over existing approaches.
[1] - T. Heskes. Empirical Bayes for learning to learn. 2000.
[2] - E. Grant, C. Finn, S. Levine, T. Darrell, and T. Griffiths. Recasting gradient-based meta-learning as hierarchical Bayes. 2018.
[3] - C. Finn, K. Xu, and S. Levine. Probabilistic model-agnostic meta-learning. 2018.
[4] - T. Kim, J. Yoon, O. Dia, S. Kim, Y. Bengio, and S. Ahn. Bayesian model-agnostic meta-learning. 2018.
[5] - J. Gordon, J. Bronskill, M. Bauer, S. Nowozin, and R. Turner. Decision-theoretic meta-learning: versatile and efficient amortization of few-shot learning. 2018.
[6] - M. Garnelo, J. Schwarz, D. Rosenbaum, F. Viola, D. J. Rezende, S. Eslami, and Y. W. Teh. Neural processes. 2018.
[7] - H. Edwards, and A. Storkey. Towards a neural statistician. 2016.


MergedFile
We study the problem of learning representations of sequence data. Recent work has built on variational autoencoders to develop variational recurrent models for generation. Our main goal is not generation but rather representation learning for downstream prediction tasks. Existing variational recurrent models typically use stochastic recurrent connections to model the dependence among neighboring latent variables, while generation assumes independence of generated data per time step given the latent sequence. In contrast, our models assume independence among all latent variables given non-stochastic hidden states, which speeds up inference, while assuming dependence of observations at each time step on all latent variables, which improves representation quality. In addition, we propose and study extensions for improving downstream performance, including hierarchical auxiliary latent variables and prior updating during training. Experiments show improved performance on several speech and language tasks with different levels of supervision, as well as in a multi-view learning setting.

This is largely an experimental paper, proposing and evaluating various modifications of variational recurrent models towards obtaining sequence data representations that are effective in downstream tasks. The highlighted contribution is a "stochastic generation" training procedure in which the training objective evaluates the reconstruction of output sequence elements from individual latent variables independently. The main claim is that the resulting model, augmented with prior updating and/or hierarchical latent variables, improves results w.r.t. the baselines.
My main concern is that the various choices are not motivated well, e.g. with examples or detailed descriptions of the issues addressed and that the resulting implications are not discussed in detail (see detailed comments below). This could perhaps be alleviated during the rebuttal discussion.
Empirically, when used in conjunction with prior updating and/or hierarchical latent variables, the proposed "stochastic generation" approach improves upon the baselines, but not when used in isolation. This is OK, but it weakens the contribution since it's more unclear what the exact advantage "stochastic generation" is, how it takes advantage of prior updating, and so on. Could you maybe discuss this in the rebuttal? The fact that not all model variants considered are evaluated on all settings also contributes to this problem (again, see below).
General questions:
- "dependence of observations at each time step on all latent variables": Unfortunately, this means that the complexity of evaluating the model during training is O(n^2), where n is the sequence size, rather than linear in the standard case. Is that correct? I think this is what is alluded to on the top on page 4. Could you discuss this trade-off?
- regarding section 2.1.: Multi-modal marginal probabilities are also used due their increased modeling power, and this again seems like a potential limitation of the proposed approach w.r.t. the baseline, and is not discussed.
- "the mean of z_t may have very small probability and thus may not be a good choice": I think this statement requires more context. The mean of z_t can have low probability in both cases (e.g. if the posterior has a high variance). Are you suggesting that the low probability issue is exacerbated by to the sampling of previous z_{t-1}? Or are you comparing to the case where the mean z_{t-1} is used instead of sampling as well?
Stochastic generation:
- While I understand where it's coming from, the term "stochastic generation" is somewhat misleading, since stochasticity is already present in the generation process for VAEs;
- Stochastic generation is introduced as a way to approximate the generation process. However, when it's introduced, it's not clear what the generation process that needs to be approximated is. Introducing the model in eq. (6-7), motivating its use and then showing how it is obtained through stochastic generation second would improve the clarity of the paper.
- Related to the point above, the implications of using the model in eq. (6-7) are not discussed. The graphical model in Figure 1 suggests that x_k depends jointly on all the (z_t)_{t=1 ... sequence_size}. Instead, in eq. (6-7), each x_k is generated independently from each z_t (for t = 1 ... T, and k sampled from a distribution which depends on t). In particular, if I understand this correctly, the distribution p(x_k | z) = p(x_k | z_1 \dots z_T) factorizes as p(x_k | z_1) p(x_k | z_2) ... p(x_k | z_T). Could you motivate this choice and its expected effect? It seems to me that this encourages each z_t to capture all the information needed to reconstruct each x_k in the corresponding window.
Experimental results:
- Table 2: I think this table since it includes most models, but it still misses RecRep (without delta = 0) and StocCon. Could you confirm whether StocCon vs. RecRep have the same setting except the use of recurrent stochastic connections in StocCon vs. using eq. (4) in RecRep with window size 1?
- In Table 4, the difference between line 5 and line 6 is interesting and I wish it was discussed more, maybe used in the visualization experiment to show how/why "stochastic generation" with a larger window improves performance.
- Figure 3, could it be that the use of hierarchical latent variables (H) accounts for the visual difference? Is a difference still observed when comparing lines 3 and 7 in Table 4, whose settings seem more comparable?
-
Minor issues:
- the lack of parenthesis around citations makes the text hard to follow at times (maybe use \citep whenever the citation mixes with the text?);
- typo: "for use in a downstream tasks"
- typo: "with graphical model as described" => "with the/a graphical model as described"


We propose Noisy Information Bottlenecks (NIB) to limit mutual information between learned parameters and the data through noise. We show why this benefits generalization and allows mitigation of model overfitting both for supervised and unsupervised learning, even for arbitrarily complex architectures. We reinterpret methods including the Variational Autoencoder, β-VAE, network weight uncertainty and a variant of dropout combined with weight decay as special cases of our approach, explaining and quantifying regularizing properties and vulnerabilities within information theory.

This paper studies "Noisy Information Bottlenecks". The overall idea is that, if the mutual information between learned parameters and the data is limited, then this prevents overfitting. It proposes to create a "bottleneck" to limit the mutual information. Specifically, the bottleneck is created by having the data depend on a noisy version of the parameters, rather than the true parameters and invoking the information processing inequality. The paper gives an example of Gaussian mean field inference. Ultimately, the analysis boils down to looking at a signal-to-noise ratio of the algorithm, which looks very much like regularization.
I think this is a very interesting direction, but the present paper is somewhat unclear. In particular, the example in section 3.1 says that a noisy information bottleneck is introduced, but then says that the modified and unmodified models have "training algorithms that are exactly equivalent." I think this example needs to be clarified. Many of the parameters here are also unclear and not properly defined/introduced. What is the relationship between $\theta$ and $\tilde\theta$ exactly? In this simple model, can we not calculate the mutual information directly (i.e., without the bottleneck)?
The connection between mutual information and generalization has been studied in several contexts [see, e.g., the references in this paper and https://arxiv.org/abs/1511.05219 https://arxiv.org/abs/1705.07809 https://arxiv.org/abs/1712.07196 https://arxiv.org/pdf/1605.02277.pdf https://arxiv.org/abs/1710.05233 https://arxiv.org/pdf/1706.00820.pdf ] and further exploration is desirable. This paper is giving an information-theoretic perspective on existing variational inference methods. Such a perspective is interesting, but needs to be further developed and explained. Specifically, how can mutual information in this context be formally linked to generalization/overfitting? Also, the definition of mutual information used in this paper uses the inferred distribution q (e.g., in eq. 2), which is somewhat unusual. As a result, constraining the model will alter the mutual information and I think the effect of this should be remarked on.
Overall, I think this paper has some interesting ideas, but those need to be fleshed out and clearly explained in a future revision.


Partially observable Markov decision processes (POMDPs) are a widely-used framework to model decision-making with uncertainty about the environment and under stochastic outcome. In conventional POMDP models, the observations that the agent receives originate from fixed known distribution. However, in a variety of real-world scenarios the agent has an active role in its perception by selecting which observations to receive. Due to combinatorial nature of such selection process, it is computationally intractable to integrate the perception decision with the planning decision. To prevent such expansion of the action space, we propose a greedy strategy for observation selection that aims to minimize the uncertainty in state. We develop a novel point-based value iteration algorithm that incorporates the greedy strategy to achieve near-optimal uncertainty reduction for sampled belief points. This in turn enables the solver to efficiently approximate the reachable subspace of belief simplex by essentially separating computations related to perception from planning. Lastly, we implement the proposed solver and demonstrate its performance and computational advantage in a range of robotic scenarios where the robot simultaneously performs active perception and planning.

This work addresses the problem of decomposing the observation acquisition from action planning in POMDPs.  Unfortunately, the paper has two major weaknesses.  First it is hindered by a confusing motivation, and the lack of clarity on the real purpose of the work is a problem throughout. Second the experiments are insufficient given current standards in the literature.
1. Motivation:  The introduction suggests that the main motivation is to reduce the computational cost of treating all observations within planning (“one must establish a trade-off between optimality and tractability”), though later, different reasons are offered (“power, processing capability, and cost constraints”).  It seems that each of these poses different constraints, and depending on which we are most concerned with, a different approximation scheme should be selected. For example, if the real concern is tractability of the POMDP solution, I don’t see why it’s not possible to just acquire all the sensor information, and afterwards decide how to approximate the tracking (this by the way is what most point-based POMDP methods effectively do).  For the proposed AP^2-POMDP, possibly a more reasonable motivation is high cost of observation acquisition; a clean argument would have to be made about the class of problems for which this constraint is crucial, why cardinality of sensor is the right way to articulate this constraint.
2. Experimental results:  The domains selected for the experiments are too simple, given current standards in the literature.  Looking at the 1D and 2D domains is fine to illustrate specific properties of your methods.  But it does not support the claim that the proposed model is more scalable than standard POMDPs.  In such simple domains, why not include results for a point-based method?  They should work in the 1D domain, probably also in the 2D domain.  Also, the setting for the 1-D domain, with a camera in every cell, seems very artificial.  First, if sensors are expensive, why put a camera in every cell?  And if they are not expensive, then why do we need to reason about which sensor to use at each step?  And why just read from k cameras at every step?  These questions point back to the concern regarding what is the real motivation for this work.  For the 2D domain, there are not even quantitative results on cumulative reward.  To be convincing, the results would need to be on substantially more complex domains; there are several POMDP benchmarks that could be considered, e.g. those in the work of Kurniawati et al.
Other comments:
-	Assumption 1 states that the observations from sensors are mutually independent give the state and action.  Can you explain why this is reasonable?  Or whether this is a strong assumption (unlikely to be met in practice)?
-	Some of the bounds seem like they could be very loose in practice, even (in the worse case) worse than the default bound of (R_max-R_min)/(1-\gamma). For example in Thm 3, in the case where the L1 distance between the 2 beliefs is 2, this is worse than the default bound.  Did you check what is the bound for the domains in the experiments?  Is it tighter than this?
-	A key statement is on p.8: “this added complexity is significantly lower than concatening the combinatorial perception actions with the planning actions”.   It is important to support this statement, ideally with both a precise complexity analysis, and with empirical results showing the lesser performance of standard point-based methods.
Minor comment:
-	The referencing style is broken and should be fixed, in particular proper use of Author (year) in the text.
-	The derivations in the top part of p.4 (Eqn 2-4 & surrounding text) are confusing, given that these apply to a standard POMDP, whereas on the previous page your present the AP^2-POMDP model. It might be better in Sec.2 to first (briefly) introduce POMDPs, with Eqn 2-4, then introduce AP^2-POMDP in Sec.3.
-	P.5: “It is worth noting that the objective function does not explicitly depend on perception actions”. This is a confusing statement; V depends on observations through b_t.  The next sentence clarifies this, but it would be better to avoid the confusing statement.
-	Alg.2:  Add a reference beside the title (unless you claim it is new).  Maybe Pineau et al. 2003.
-	P.7: “can be combined with any sampling and pruning method in other solvers” -> Add references for such sampling & pruning methods.


In this paper, we propose a capsule-based neural network model to solve the semantic segmentation problem. By taking advantage of the extractable part-whole dependencies available in capsule layers, we derive the probabilities of the class labels for individual capsules through a recursive, layer-by-layer procedure. We model this procedure as a traceback pipeline and take it as a central piece to build an end-to-end segmentation network. Under the proposed framework, image-level class labels and object boundaries are jointly sought in an explicit manner, which poses a significant advantage over the state-of-the-art fully convolutional network (FCN) solutions. Experiments conducted on modified MNIST and neuroimages demonstrate that our model considerably enhance the segmentation performance compared to the leading FCN variant.

Based on the CapsNet concept of Sabour the authors proposed a trace-back method to perform a semantic segmentation in parallel to classification. The method is evaluate on MNIST and the Hippocampus dataset.
The paper is well-written and well-explained. Nevertheless, I think it would be useful to have some illustrations about the network architecture. Some stuff which is explained in text could be easily visualized in a flow chart. For example, the baseline architecture and your Tr-CapsNet could be easily explained via a flow chart. With the text only, it is hard to follow. Please think about some plots in the final version or in the appendix. One question which is aligned to that: How many convolutional filters are used in the baseline model?
Additionally, think about a pseudo-code for improved understandability.
Some minor concerns/ notes to the authors:
1.	At page 5: You mentioned that the parameters lambda1 and lambda 2 are important hyper-parameters to tune. But in the results you are not explaining how the parameters were tuned. So my question is: How do you tuned the parameters? In which range do you varied the parameters?
2.	Page 6; baseline model: Why do you removed the pooling layers?
3.	I’m curious about the number of parameters in each model. To have a valid discussion about your model is better than the U-Net-6 architecture, I would take into account the number of parameters. In case that your model is noticeably greater, it could be that your increased performance is just due to more parameters. As long as your discussion is without the number of parameters I’m not convinced that your model is better. A comparison between models should be always fair if two models are architectural similar.
4.	Why is the magnitude of lambda1 so different between the two dataset that you used?
5.	Could you add the inference times to your tables and discuss that in addition?
6.	What kind of noise is added to MNIST?
7.	What is the state-of-the-art performance on the Hippocampus dataset?
8.	What would be the performance in your experiments with a MaskRCNN segmentation network?
9.	I’m not familiar with the Hippocampus dataset. I missed a reference where the data is available or some explaining illustrations.
10.	For both datasets, more illustrations about the segmentation performance would be fine to evaluate your method. At least in the appendix…

My major concern is that both datasets are not dealing with real background noise. I’m concerned that the results are not transferable to other datasets and that the method shines promising just because of the simple datasets only. For example, due to the black background MNIST digits are well separated (if we skip that you added some kind of noise). So, from that point of view your results are not convincing and the discussion of your results appearing sparse and not complete.
To make your results transparent you could think about to publish the code somewhere.


Neural Networks (NN) have achieved state-of-the-art performance in many tasks within image, speech, and text domains. Such great success is mainly due to special structure design to fit the particular data patterns, such as CNN capturing spatial locality and RNN modeling sequential dependency. Essentially, these specific NNs achieve good performance by leveraging the prior knowledge over corresponding domain data. Nevertheless, there are many applications with all kinds of tabular data in other domains. Since there are no shared patterns among these diverse tabular data, it is hard to design specific structures to fit them all. Without careful architecture design based on domain knowledge, it is quite challenging for NN to reach satisfactory performance in these tabular data domains. To fill the gap of NN in tabular data learning, we propose a universal neural network solution, called TabNN, to derive effective NN architectures for tabular data in all kinds of tasks automatically. Specifically, the design of TabNN follows two principles: to explicitly leverage expressive feature combinations and to reduce model complexity. Since GBDT has empirically proven its strength in modeling tabular data, we use GBDT to power the implementation of TabNN. Comprehensive experimental analysis on a variety of tabular datasets demonstrate that TabNN can achieve much better performance than many baseline solutions.

This paper proposes a hybrid machine learning algorithm using Gradient Boosted Decision Trees (GBDT) and Deep Neural Networks (DNN). The intended research direction on tabular data is essential and promising. However, the proposed technique does not seem to be handling the problem foundationally well. It seems heavily dependent on GBDT. It also shows itself in the results that final algorithm is almost indistinguishable from GBDT regarding results. Moreover, I  don't think that the data sets in experiments are good enough to cover the importance and the nature of the problem.
Pros:
-This is a crucial line of research direction that aims to make DNNs applicable to many real-world problems (beyond speech and vision) in which discrete data and heterogeneous features exist such as engagement prediction, recommendation, and search.
-The starting point of using GBDT seems like a good choice.
-The Paper is mostly well written except occasional repetitions and missing acronym definitions.
Cons:
-The proposed technique does not seem to be original enough, and it does not handle the problem foundationally well. I do not think that there is enough justification/demonstration for the fact that a general NN solution for Tabular Data invented. The proposed technique is heavily dependent on GBDT (Indeed the algorithm and the learned trees are used at least three times). This shows itself in the results; i.e., the proposed algorithm is either negligibly performing better than GBDT or when  GBDT dependence removed, it performs worse. It seems to me that (except the minor small section of streaming data), the paper is more like a proper verification of how tree-based learning algorithms work very well in tabular data--which is far from the basis of the paper and does not make the paper novel enough for ICLR.
-The proposed technique seems to include very heavy feature engineering and several ad-hoc practical steps--that is far from the motivation of using NN in tabular data.
-In the provided benchmark data sets the depth of the analysis seems to be enough. However, in the proposed domain of tabular data, often data sets are significantly more high dimensional in reality and include at least one set of sparse large dimensional features  (e.g., unstructured raw text for the search queries.) In such scenarios, it had been showed that wide-and-deep NNs perform decently. However such problems are entirely missing in the results section. I also think that this is a lost opportunity for the authors as they could be showing that it is the NN part contributing.


Skills learned through reinforcement learning often generalize poorly across tasks and re-training is necessary when presented with a new task. We present a framework that combines techniques in formal methods with reinforcement learning (RL) that allows for the construction of new skills from existing ones with no additional exploration necessary. Our method also allows for convenient specification of complex temporal dependent tasks using logical expressions. We provide theoretical results for our composition technique and evaluate on a simple grid world simulation as well as a robotic manipulation task.

This work proposed using temporal logic formulas to augment RL learning via the composition of previously learned skills. This work was very difficult to follow, so it is somewhat unclear what were the main contributions (since much of this seems to be covered by other works as referenced within the paper and as related to similar unreferenced works below). Moreover, regarding the experiments, many things were unclear (some of the issues are outlined below). While the overall idea of using logic in this way to help with skill composition is interesting and exciting, I believe several things must be addressed with this work. This includes: situating this work more clearly against existing similar works which use logic in this way, clearly defining the novel contributions of this work as compared to those and others, overall making the methodology more clear and specific (including experimental methodology), and comparing/contrasting against (or at least discussing differences with) methods with similar motivations (e.g., HRL multi-task learning, meta-learning) to emphasize the need/importance of this work — I am aware that at least 1 HRL work is mentioned, but this work is not really contrasted against it to help situate it.
Questions/Concerns about Experiments:
+ Does Figure 5 show the averaged return over 5 runs, sum of discounted rewards averaged over 5 episodes per update step, or 5 episodes, each from a separate run averaged together? It is a bit unclear especially because the main text and the figure caption slightly differ. Also, average discounted return is somewhat different than average return,  suggest updating the label to be clear also with the discount factor used.
+ What were the standard deviations for this across experiments? Even with averaging it seems that these runs are very high variance, would be good to understand what variance bounds to expect if using this method.
+ Why were average discounted returns reported in Figure 5 and not in Table 1?
+  What were the standard deviations on success rate and training time? Also what about sample complexity?
+ To my understanding the benefit here is reusability of learned skills via the automata methods described here. It would have made sense to compare against other HRL or multi-task learning methods in addition to just SQL or learning from scratch. For example how would MAML compare to this?
+ It is also unclear whether the presented results in Table 1 and Figure 5 are on the real robot or in simulation. The main text says, “All of our training is performed in simulation and the policy is able to transfer to the real robot without further fine-tuning.” So does this mean that Figure 5 is simulated results and Table 1 is on the real robot?
Citations that should likely be made:
+ Giuseppe, Luca Iocchi, Marco Favorito, and Fabio Patrizi. "Reinforcement Learning for LTLf/LDLf Goals." arXiv preprint arXiv:1807.06333 (2018).
+ Camacho, Alberto, Oscar Chen, Scott Sanner, and Sheila A. McIlraith. "Decision-making with non-markovian rewards: From LTL to automata-based reward shaping."  In Proceedings of the Multi-disciplinary Conference on Reinforcement Learning and Decision Making (RLDM), pp. 279-283. 2017.
+ Camacho, Alberto, Oscar Chen, Scott Sanner, and Sheila A. McIlraith. "Non-Markovian Rewards Expressed in LTL: Guiding Search Via Reward Shaping." In Proceedings of the Tenth International Symposium on Combinatorial Search (SoCS), pp. 159-160. 2017.
Typos/Suggested grammar edits:
“Skills learned through (deep) reinforcement learning often generalizes poorly across tasks and re-training is necessary when presented with a new task.” —> Often generalize poorly
“We present a framework that combines techniques in formal methods with reinforcement learning (RL) that allows for convenient specification of complex temporal dependent tasks with logical expressions and construction of new skills from existing ones with no additional exploration.” —> Sentence kind of difficult to parse and is a run-on
“Policies learned using reinforcement learning aim to maximize the given reward function and is often difficult to transfer to other problem domains.” —> ..and are often..
“by authors of (Todorov, 2009) and (Da Silva et al., 2009)” —> by Todorov (2009) and Da Silva et al. (2009) Also several other places where you can use \citet instead of \cite


NEURAL NETWORK GRADIENT-BASED LEARNING OF BLACK-BOX FUNCTION INTERFACES
Deep neural networks work well at approximating complicated functions when provided with data and trained by gradient descent methods. At the same time, there is a vast amount of existing functions that programmatically solve different tasks in a precise manner eliminating the need for training. In many cases, it is possible to decompose a task to a series of functions, of which for some we may prefer to use a neural network to learn the functionality, while for others the preferred method would be to use existing black-box functions. We propose a method for end-to-end training of a base neural network that integrates calls to existing blackbox functions. We do so by approximating the black-box functionality with a differentiable neural network in a way that drives the base network to comply with the black-box function interface during the end-to-end optimization process. At inference time, we replace the differentiable estimator with its external blackbox non-differentiable counterpart such that the base network output matches the input arguments of the black-box function. Using this "Estimate and Replace" paradigm, we train a neural network, end to end, to compute the input to blackbox functionality while eliminating the need for intermediate labels. We show that by leveraging the existing precise black-box function during inference, the integrated model generalizes better than a fully differentiable model, and learns more efficiently compared to RL-based methods.

This paper presents an approach, called EstiNet, to train a hybrid models which uses both neural networks and black-box functions. The key idea is that, during training, a neural network can be used to approximate the functionality of the black-box functions, which makes the whole system end-to-end differentiable. At test time, the true black-box functions are still used. The training objective composes two parts: L_bbf, the loss for approximating the black-box function and L_target, the loss for the end-to-end goal. They tried different variations of when to train the black-box function approximator or not. It is shown to outperform the baselines like end-to-end differentiable model or NALU over 4 synthetic tasks in sample efficiency. There are some analysis about how the entropy loss and label smoothing helps with the gradient flow.
The proposed model is interesting, and is shown to be effective in the synthetic tasks. The paper is well-written and easy to follow. However, some of the experiment details are missing or scattered in the text, which might make it hard for the readers to reproduce the result. I think it helps to have the experimental details (number of examples, number of offline pretraining steps, size of the neural network, etc) organized in some tables (could be put in the appendix).
Two main concerns about how generally applicable is the proposed approach:
1. It helps to show how L_target depends on L_bbf, or how good the approximation of the black-box function has to be to make the approach applicable. For example, some functions, such as sorting, are hard to approximate by neural network in a generalizable way, so in those cases, is it still possible to apply the proposed approach?
2.The proposed approach can be better justified by discussing some potential real world applications. Two closely related applications I can think of are visual question answering and semantic parsing. However, it is hard to find good black-box functions for VQA and people often learn them as neural networks, and the functions in semantic parsing often need to interact with a database or knowledge graph, which is hard to approximate with a neural network.
Some minor issues:
Table 3 isn’t very informative since k=2 and k=3 provides very similar results. It would help to show how large k needs to be for the performance to severely degrade.
Missing references:
The Related Works section only reviewed some reinforcement learning work on synthetic tasks. However, with some bootstrapping, RL is also shown to achieve state-of-the-art performance on visual question answering and semantic parsing tasks (Johnson et al, 2017; Liang et al, 2018), which might be good to include here.
Johnson, J., Hariharan, B., van der Maaten, L., Hoffman, J., Fei-Fei, L., Zitnick, C. L., & Girshick, R. B. (2017, May). Inferring and Executing Programs for Visual Reasoning. In ICCV (pp. 3008-3017).
Liang, C., Norouzi, M., Berant, J., Le, Q., & Lao, N. (2018). Memory augmented policy optimization for program synthesis with generalization. arXiv preprint arXiv:1807.02322.


Motivated by the study of generalization in biological intelligence, we examine reinforcement learning (RL) in settings where there are information-theoretic constraints placed on the learner's ability to represent a behavioral policy. We first show that the problem of optimizing expected utility within capacity-limited learning agents maps naturally to the mathematical field of rate-distortion (RD) theory. Applying the RD framework to the RL setting, we develop a new online RL algorithm, Capacity-Limited Actor-Critic, that learns a policy that optimizes a tradeoff between utility maximization and information processing costs. Using this algorithm in a 2D gridworld environment, we demonstrate two novel empirical results. First, at high information rates (high channel capacity), the algorithm achieves faster learning and discovers better policies compared to the standard tabular actor-critic algorithm. Second, we demonstrate that agents with capacitylimited policy representations avoid 'overfitting' and exhibit superior transfer to modified environments, compared to policies learned by agents with unlimited information processing resources. Our work provides a principled framework for the development of computationally rational RL agents.

(score raised from 6 to 7 after the rebuttal)
The paper explores the application of the rate-distortion framework to policy learning in the reinforcement learning setting. In particular, a policy that maps from states to actions is considered an information theoretic channel of limited capacity. This viewpoint provides an interesting angle which allows modeling/learning of (computationally) bounded-rational policies. While capacity-limitation might intuitively seem to be a disadvantage, intriguing arguments (based on solid theoretical foundations, rooted in first principles) can be made in favor of capacity-limited systems. Two of the main-arguments are that capacity-limited policies should be faster to learn and be more robust, i.e. generalize better. After thoroughly introducing these arguments on a less formal level and putting them into perspective with regard to reinforcement learning and related work in the literature, the paper demonstrates these properties in a toy grid-world example. When compared against a vanilla actor-critic (AC) algorithm, the capacity-limited version is shown to converge faster and reach better final policies. The paper then extends the basic version of the algorithm, which requires knowledge of the optimal value function, towards simultaneously learning the value function. While any theoretical guarantees are lost, the empirical results are still in line with the theoretical benefits, outperforming vanilla AC and producing better results in previously unencountered variations of the grid-world environment.
The paper is very well written and the toy-examples illustrate the theoretical advantages in a very nice and intuitively understandable way. The topic of modeling capacity-limited RL agents and exploring how capacity-limitation is an advantage, rather than a “bug” is very timely and important. In particular, rate-distortion theory might provide key-insights into building agents that generalize well, which is among the major open problems in reinforcement learning. The paper is thus very timely and highly relevant to a broad audience.
The main weakness of the paper is that it is of quite limited novelty and that the brute-force approach towards using Blahut-Arimoto in RL is unlikely to scale to large, complex state-/action-spaces without major additional work. Continuous state-/action-spaces are in principle covered by the theory, but they come with additional caveats and subtleties (I appreciate the authors using discrete notation with sums instead of integrals). Additionally, when simultaneously learning the value function (in the online setting), any guarantees about Blahut-Arimoto convergence are lost. However, solving either of these issues is hard and many attempts have been made in the communications community. Despite these weaknesses I argue for accepting and presenting the paper at the conference for the following reasons:
- modelling capacity-limited agents via ideas from rate-distortion theory (which is very closely related to free-energy optimization, such as ELBO maximization, Bayesian inference and the MDL principle) is an underrated topic in reinforcement learning. On a conceptual level, the strong idea is that moving away from strict optimization and infinite-capacity systems is not a shortcoming but can actually help building agents that perform better and generalize better. This is not a well established idea in the community. The paper does a good job at introducing the general idea, illustrating it intuitively with toy examples and pointing out relevant literature.
- Simultaneously learning the value function is necessary in the RL setting, but breaks quite a bit of the theory. However, very similar ideas seem to work quite well empirically in other settings, such as for instance ELBO maximization in VAEs, where the “value function” is the log-likelihood (under the decoder), which is learned simultaneously while learning a “policy” (the encoder) under capacity limitation (the KL term). Similar arguments can be made for modern InfoBottleneck-style objectives in deep learning. Based on this empirical observation, it is not unlikely that simultaneous learning of the value function works reasonably well without catastrophically collapsing in other settings and tasks.
- While achieving a solution that strictly lies on the rate-distortion curve might be crucial in communications, it might be of lesser significance for building RL agents that generalize well - slight sub-optimalities (solutions that lie off the RD curve) should still yield interesting agents. Therefore, losing theoretical guarantees might be less severe for simply exploring how much the idea can be scaled up empirically.
Minor issues:
1) While the paper, strictly speaking, introduces a novel algorithm and the Bellman loss function (which requires knowledge of the optimal value function), I think that the main contribution is a clear and well-focused introduction of rate-distortion theory in the context of RL, including very illustrative toy examples. I do consider this an important contribution.
2) Transfer to novel environments. The final example (Fig. 4) does show that the capacity limited agent performs better in novel environments. However, I’m not entirely convinced that this demonstrates “superior transfer to novel environments” (from the abstract). While the latter might very well arise from capacity-limitations, I think that in the example in the paper there is not too much transfer going on, but the capacity-limited agent simply has a more stochastic policy which helps if unknown walls are in the way. After all, the average accumulated reward of the capacity limited agent does also decrease significantly in the novel environment - it simply does a slightly better random walk than the AC (correct me if I’m wrong, of course). On page 7, last paragraph this is phrased as: “agents retain knowledge of exploratory actions”. In my opinion this wording is a bit too strong to simply describe increased stochasticity.
3) Since the paper does provide a good overview over the literature, I think it would help to mention that the current main approach towards generalizing (deep) RL is via hierarchical RL (options framework, etc) and provide a good reference.
4) At the very end of the intro you might also want to mention that rate-distortion has been used before in the context of decision-making (not RL), for instance under the term rational inattention.
5) Page 5, last paragraph: the paper mentions that one Blahut-Arimoto iteration is enough. This is an empirical observation, justified by the toy experiments. However, the wording sounds like this is a generally known fact. Please rephrase to emphasize that this must not necessarily hold true in general and that convergence behavior might crucially depend on this.
6) It would be good to give readers some guidance towards choosing beta if doing an exhaustive grid-search is infeasible. I am aware that there is no good general rule or recipe, but perhaps something can be added to the discussion (even if it is just mentioning that there is no good heuristic, etc. - however, there should be plenty of research in communications that deals with estimating the RD curve from as few points as possible).
7) Please consider adding this reference - it has a very similar objective function (but for navigating towards multiple goals) and is very much in line with some of the theoretical arguments.
Informational Constraints-Driven Organization in Goal-Directed Behavior - Van Dijk, Polani, 2013.


FEATURE-WISE BIAS AMPLIFICATION
We study the phenomenon of bias amplification in classifiers, wherein a machine learning model learns to predict classes with a greater disparity than the underlying ground truth. We demonstrate that bias amplification can arise via an inductive bias in gradient descent methods that results in the overestimation of the importance of moderately-predictive "weak" features if insufficient training data is available. This overestimation gives rise to feature-wise bias amplificationa previously unreported form of bias that can be traced back to the features of a trained model. Through analysis and experiments, we show that while some bias cannot be mitigated without sacrificing accuracy, feature-wise bias amplification can be mitigated through targeted feature selection. We present two new feature selection algorithms for mitigating bias amplification in linear models, and show how they can be adapted to convolutional neural networks efficiently. Our experiments on synthetic and real data demonstrate that these algorithms consistently lead to reduced bias without harming accuracy, in some cases eliminating predictive bias altogether while providing modest gains in accuracy.

Summary:
In this paper the authors identify a specific source of marginal class probability bias that occurs when using logistic regression models. Using synthetic and real datasets they demonstrate this bias and explore characteristics of the data that exacerbate the issue. Finally, they propose two methods for correcting this bias in logistic regression models and neural network models with logistic output layers and evaluate these methods on several benchmark datasets.
Review:
Overall, I found the paper well-written, the problem well-motivated, and the proposed methods clear and reasonable. While I have a few concerns about presentation and experimentation, these are issues that can easily be remedied and I recommend acceptance.
Major comments:
- The authors repeatedly say that gradient descent is the cause of the bias amplification (e.g. Section 2.2 title, "...features that are systematically overestimated by gradient descent.", "... i.e., a consequence of gradient descent's inductive bias.", "... gradient descent may lead to systematic bias..."). The inductive bias they describe is coming from the use of logistic regression, not the use of gradient descent. Specifically, a logistic regression model has a convex likelihood, which means that regardless of what algorithm is used to maximize the likelihood, it should converge to the same point. In fact, most off-the-shelf implementations of logistic regression do not use vanilla gradient descent. Further, gradient descent may be used to estimate the parameters of any number of models which may or may not have the same inductive bias the authors describe.
- I thought the related work section was well-written and would strongly recommend moving it to the beginning of the paper as it motivates the entire problem. I also think it could be helpful to ground the technical definitions of bias amplification in a meaningful example.
- I think that the experimental setup for comparing \ell_1 regularization to the proposed feature selection methods is not quite fair. In particular, the hyperparameters of the "expert" method are selected to minimize bias subject to the constraint that loss not increase. In contrast, the \ell_1 regularization hyperparameter is selected purely to minimize bias. Instead, I would select the \ell_1 regularization hyperparameter in the same way as the expert method, that is, to minimize bias subject to a constraint on loss. In general, I think hyperparameters should be selected using the same criterion for all methods.
- The authors make a point of highlighting results on the "prostate" which showed a large increase in accuracy along with a large decrease in bias. I think the paper would benefit from some exploration of why this happened. Specifically, it would be valuable to answer the question: what are the properties of the "prostate" dataset that make this method so effective and are these properties general and identifiable a priori?
- Section 2, paragraph 2, line 5: The stated goal in this paragraph is "minimizing 0-1 loss on unknown future i.i.d. samples". As stated in the introduction, this is, in fact, not the goal. The goal is to minimize loss while also minimizing bias. A larger criticism that I would have of this work is: if minimizing bias is a first order goal, then why are we using empirical risk minimization in the first place? Put another way, why use post-hoc correction for an objective function that does not match our actual stated goals rather than using an objective function that does?
Minor comments:
- Section 1, paragraph 4, line 2: "Weak" is not clearly defined here. Is it different than "moderately-predictive"?
- Section 2.1, last paragraph, line 1: I understand what the authors are saying when they say "Bias amplification is unavoidable", but it is avoidable by changing our objective function. I would consider rewording this statement to something like "Using an ERM objective will lead to bias amplification when the learning rule..."
- Equation 4: I believe h should be changed to f in this equation.
- Equation 6: L is not defined anywhere.
- Table 1: As defined in equation 1, B_D(h_s) should be between 0 and 1. Also, the accuracy results for the glioma dataset have the wrong result in bold.
- Section 4, methodology paragraph, line 5: forthe --> for the
- Section 5, paragraph 5, lines 5-6: Feature selection is not used "only to improve accuracy". For example, Kim, Shah, and Doshi-Valez (2015) use feature selection to improve interpretability (https://beenkim.github.io/papers/BKim2015NIPS.pdf).


For distributions p and q with different support, the divergence D(p||q) generally will not exist. We define a spread divergenceD(p||q) on modified p and q and describe sufficient conditions for the existence of such a divergence. We give examples of using a spread divergence to train implicit generative models, including linear models (Principal Components Analysis and Independent Components Analysis) and non-linear models (Deep Generative Networks).

Pros:
- interesting idea.
Cons:
- the paper forgets the state of the art for comparisons (optimal transport, data processing inequalities)
- the paper formally shows little as most results are in fact buried in the text and it is hard to tell the formal from the informal.
- experiments fall short of really using the setting proposed
- the paper focuses too much on keeping the identity of the indiscernibles and forgets the study of other properties (including downsides, such as variance increase)
Detail:
* The paper claims to propose a "theory" for spread divergences (conditioning a f-divergence by a "third-party" conditional distribution on supports which makes supports match) still keeping the identity of indiscernibles.
* The paper recycles the notion of Spread f-divergences from Zhang et al. (which makes a circular reference to this paper for the introduction of these divergences).
* The paper motivates the notion of spread divergences by the fact that f-divergences impose matching supports (Section 1), not mentioning that optimal transport theory is a much natural fit for any such kind of setting (e.g. Wasserstein distances). This is a big omission and a missed occasion for a potentially interesting discussion.
* The paper then claims that "spread noise makes distributions more similar" (Section 2.1), not mentioning that equation (8), which it claims to have been shown by Zhang et al. paper (see below), is in fact a data processing inequality *long known*. They will find it, along with a huge number of other useful properties, in series of IEEE T. IT papers, among which Pardo and Vajda's "About distances of discrete distributions satisfying the data processing theorem of information theory",  Van Erven and Harremoes, "Re ́nyi Divergence and Kullback-Leibler Divergence" (for the KL / Rényi divergence, but you have more references inside), etc. .
* The paper then goes on "showing" (a word used often, even when there is not a single Theorem, Lemma or the like ever stated in the paper...) several properties (Section 2.2). The first states that (9) is equivalent to P being invertible. It is wrong because it is just in fact stating (literally) that P defines an injective mapping. The second states that (11) is equivalent to (12), without the beginning of a proof. I do need to see a proof, and in particular how you "define" an invertible transform "p^-1".
* The paper then gives two examples (Sections 3, 4). In Section 3, I am a bit confused because it seems that p and q must have supports in IR, which limits the scope of the example. The same limitation applies to Section 4, even when it is a bit more interesting. In all cases, the authors must properly state a Lemma in each Section that states and shows what is claimed before Section 3.
* The paper then makes several experiments. Unless I am mistaken, it seems that Section 5.1 relies on a trick that does not change the support from x to y. Therefore, what is the interest of the approach in this case ? In Section 5.2, isn’t the trick equivalent to considering ICA with a larger \gamma ?
* A concern is that the paper says little about the reason why we should pick one p(y|x) instead of another one. The focus is on the identity of indiscernibles. The paper also forgets some potential drawbacks of the technique, including the fact that variance increases — the increase can be important with bad choices, which is certainly not a good thing.


MULTI-DOMAIN ADVERSARIAL LEARNING
Multi-domain learning (MDL) aims at obtaining a model with minimal average risk across multiple domains. Our empirical motivation is automated microscopy data, where cultured cells are imaged after being exposed to known and unknown chemical perturbations, and each dataset displays significant experimental bias. This paper presents a multi-domain adversarial learning approach, MULANN, to leverage multiple datasets with overlapping but distinct class sets, in a semisupervised setting. Our contributions include: i) a bound on the average-and worst-domain risk in MDL, obtained using the H-divergence; ii) a new loss to accommodate semi-supervised multi-domain learning and domain adaptation; iii) the experimental validation of the approach, improving on the state of the art on three standard image benchmarks, and a novel bioimage dataset, CELL.

Summary:
The manuscript proposes a multi-domain adversarial learning (MDL) method called MULANN, to leverage multiple datasets with overlapping but distinct class sets, in a semi-supervised setting. The authors define a new discrimination task to discriminate, within each domain, labeled samples from unlabeled ones that most likely belong to extra classes (classes with no labeled or unlabeled samples in the domain). They also introduce a bound on the average- and worst-domain risk in MDL, obtained using the H-divergence.
Strengths:
- The idea of using discriminators for separating the labeled samples from unlabeled ones that most likely belong to extra classes is interesting.
- A new generalization bound for MDL is introduced.
- The paper was clear, well written, well-motivated and nicely structured.
- The authors perform numerous empirical experiments on several types of problems on various datasets (Digit, OFFICE,CELL) successfully showing how the MULANN can reduce the nasty effects of the adversarial domain discriminator and repulse (a fraction of) unlabeled examples from labeled ones in each domain.
Weaknesses:
- all the experiments except the last row of Table 2 concern adaptation between two domains. Given the paper title, the reviewer would have expected more experiments in a multiple domain context. More precisely, for the digit datasets, the reviewer was interested to see how the proposed MDL performs on jointly adapting SVHN, MNIST, MNIST-M, and USPS or jointly adapting DSLR, Amazon, and Webcam for OFFICE dataset. Moreover, comparison with some of the DA baselines (ADDA[1], DSN[2]) is missing.
- The authors propose to rank the unlabeled samples of each domain according to the entropy of their classification of the current classifier. Obviously there must be some false ranking (specially at the initial stages of updating the classifier) for the unlabeled samples (e.g. the classifier may output high entropy for the unlabeled samples of the classes with labeled samples) and they may harm the performance of adaptation.  It is not clear how MULANN can work in this situation and how its performance vary with the noisy signals conveyed in those false pseudolabeled samples.
- Although the paper introduces the generalization bound for MDL, it does not give new formulation or algorithm to handle MDL (MULANN handles only the class asymmetry when domains involve distinct sets of classes and it has nothing to do with MDL). hence, there is no connection between the theoretical results on MDL generalization bound and the proposed method MULANN.
- Since each domain may have different number of classes, it is not clear how the number of classes (L) is set in the classification module (maximum number of classes in all domain?).
The reviewer is also interested to see how the the generalization bound introduced in this paper is related to the recent theoretical works [3],[4] on MDL.
[1] Tzeng, Eric, et al. "Adversarial discriminative domain adaptation." Computer Vision and Pattern Recognition (CVPR). Vol. 1. No. 2. 2017.
[2] Bousmalis, Konstantinos, et al. "Domain separation networks." Advances in Neural Information Processing Systems. 2016.
[3] Zhao, Han, et al. "Multiple Source Domain Adaptation with Adversarial Learning." Advances in Neural Information Processing Systems. 2018.
[4] Hoffman, Judy, Mehryar Mohri, and Ningshan Zhang. "Algorithms and Theory for Multiple-Source Adaptation."  Advances in Neural Information Processing Systems. 2018.


Neural text generation models such as recurrent networks are typically trained by maximizing data log-likelihood based on cross entropy. Such training objective shows a discrepancy from test criteria like the BLEU metric. Recent work optimizes expected BLEU under the model distribution using policy gradient, while such algorithm can suffer from high variance and become impractical. In this paper, we propose a new Differentiable Expected BLEU (DEBLEU) objective that permits direct optimization of neural generation models with gradient descent. We leverage the decomposability and sparsity of BLEU, and reformulate it with moderate approximations, making the evaluation of the objective and its gradient efficient, comparable to common cross-entropy loss. We further devise a simple training procedure with ground-truth masking and annealing for stable optimization. Experiments on neural machine translation and image captioning show our method significantly improves over both cross-entropy and policy gradient training.

Differentiable Expected BLEU for Text Generation
Paper Summary:
Neural translation systems optimizes training data likelihood, not the end metric of interest BLEU. This work proposes to approximate BLEU with a continuous, differentiable function that can be optimized during training.
Review:
The paper reads well. It has a few but crucial missing references. The motivation is easy to understand and a relevant problem to work on. The main weaknesses of the work lies in its very loose derivations, and its weak empirical results.
First on context/missing references: the author ignores approaches optimizing BLEU with log linear models (Franz Och 2003), and the structured prediction literature in general, both for exact (Tsochantaridis et al 2004) and approximate search (Daume and Marcu 2005). This type of approach has been applied to NMT recently (Edunov et al 2018). Your paper also misses important references addressing BLEU optimization with reinforcement strategies (Norouzi et al 2016) or (Bahdanau et al 2017). Although not targeting BLEU directly (Wiseman and Rush 16) is also a reference to cite wrt optimizing search quality directly.
On empirical results, you chose to work IWSLT in the de-en direction while most of the literature worked on en-de. It prevents comparing your results to other papers. I would suggest to switch directions and/or to report results from other methods (Ranzato et al 2015; Wiseman and Rush 2016; Norouzi et al 2016; Edunov et al 2018). De-en is generally easier than en-de (generating German) and your BLEU scores are particularly low < 25 for de-en while other methods ranges in 26-33 BLEU for en-de (Edunov et al 2018).
On the method itself, approximating BLEU with a continuous function is not easy and the approach you take involves swapping function composition and expectation multiple times in a loose way. You acknowledge that (7) is unprincipled but (10) is also problematic since this equation does not acknowledge that successive ngrams overlap and cannot be considered independent. Also, the dependence of successive words is core to NMT/conditional language models and the independence hypothesis from the footnote on page 4 can be true only for a bag of word model. Overall, I feel that given the shortcuts you take, you need to justify that your approximation of BLEU is still correlated with BLEU. I would suggest to sample from a well trained NMT system to collect several hypotheses and to measure how well your BLEU approximation correlate with BLEU. How many times BLEU decides that hypA > hypB but your approximation invert this relation? is it true for large difference, small difference of BLEU score? at low BLEU score, high BLEU score?
Finally, you do not mention the distinction between expected BLEU  \sum_y P(y|x) BLEU(y, ref) and the BLEU obtained by beam search which only look at (an estimate of) the most likely sequence y* = argmax P(y|x) . Your approach and most reinforcement strategy targets optimizing expected BLEU, but this has no guarantee to make BLEU(y*, ref) any better. Could you report both an estimate of expected BLEU and beam BLEU for different methods? In particular, MERT (), beam optimization (Wiseman and Rush 2016) and  structured prediction (Edunov et al 2018) explicitly make this distinction. This is not a side issue as this discussion is in tension with your motivations.
Review Summary:
The paper misses important references. It chooses an empirical setup which prevents comparison with related work, and the report results on de-en seem weak. The proposed approach does not bound or estimate how far from BLEU is the proposed approximation. This means that the authors need to justify empirically that it preserves correlation with BLEU, which is not shown in the paper.
Missing references
An Actor-Critic Algorithm for Sequence Prediction (ICLR 2017)  Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, Yoshua Bengio
Hal Daume III and Daniel Marcu. Learning as search optimization: Approximate large margin methods for structured prediction. ICML 2005.
Sergey Edunov, Myle Ott, Michael Auli, David Grangier, Marc'Aurelio Ranzato
Classical Structured Prediction Losses for Sequence to Sequence Learning, NAACL 18
Minimum Error Rate Training in Statistical Machine Translation Franz Josef Och. 2003 ACL
I. Tsochantaridis, T. Hofmann, T. Joachims, and Y. Altun, Support Vector Machine Learning for Interdependent and Structured Output Spaces, ICML 2004.
Mohammad Norouzi, Samy Bengio, Zhifeng Chen, Navdeep Jaitly, Mike Schuster, Yonghui Wu, Dale Schuurmans, Reward Augmented Maximum Likelihood for Neural Structured Prediction, 2016
Sequence-to-Sequence Learning as Beam-Search Optimization, Sam Wiseman and Alexander M. Rush., EMNLP 2016


Generative neural networks map a standard, probability distribution to a complex high-dimensional distribution, which represents the real world data set. However, a determinate input distribution as well as a specific architecture of neural networks may impose limitations on capturing the diversity in the high dimensional target space. To resolve this difficulty, we propose a training framework that greedily produce a series of generative adversarial networks that incrementally capture the diversity of the target space. We show theoretically and empirically that our training algorithm converges to the theoretically optimal distribution, the projection of the real distribution onto the convex hull of the network's distribution space.

The paper introduces an incremental training method for GAN's for capturing the diversity of the input space. The paper demonstrate that the proposed method allows smaller distances between the true and generated distribution. I find the idea interesting, but fear that the 60-100 small ensemble models could be replaced by a larger model.
I am curious about why we need incremental training when it seems like we could directly train all the networks jointly. The corresponding generative model is simply stronger so all the convergence arguments would still hold. Is the statistical distance a reasonable estimate for you to determine whether you need an additional generator for incremental training?
Also what are the generator architectures for the experiments? How can you put 60-100 generators within the GPU memory? The latent variable dimension seem to be only 1 for each of your generator? That seems to be seriously handicapping the capacity of each individual generator (to just some data points), so the ensemble distribution might be obtained simply by using a larger dimension z?
There are also other measurements that are used by the GAN community, such as inception score, FID score and samples. It seems also reasonable to verify the effectiveness of this method on CIFAR or LSUN datasets, where the method would have a greater improvement because the data distributions are more complex.
Minor points:
- How do you measure the "Wasserstein distance" for high-dimensional distributions?
- What not set $\omega_i$ to be always 1? The subsampling process introduced in Algorithm 2 seem to enforce this, and you do this for all the experiments.
- Fix citation typos.
- Fix \mathbf for vector quantities, such as x and z.
- Since the generative models have the same architecture, does the non-convex argument becomes moot when you have a mixture of 2 generators?


Neural machine translation (NMT) systems have reached state of the art performance in translating text and are in wide deployment. Yet little is understood about how these systems function or break. Here we show that NMT systems are susceptible to producing highly pathological translations that are completely untethered from the source material, which we term hallucinations. Such pathological translations are problematic because they are are deeply disturbing of user trust and easy to find with a simple search. We describe a method to generate hallucinations and show that many common variations of the NMT architecture are susceptible to them. We study a variety of approaches to reduce the frequency of hallucinations, including data augmentation, dynamical systems and regularization techniques, showing that data augmentation significantly reduces hallucination frequency. Finally, we analyze networks that produce hallucinations and show that there are signatures in the attention matrix as well as in the hidden states of the decoder.

My major concern about the work is that the studied model is quite weak.
	"All models we present are well trained with a BLEU score of at least 20.0 on the test set, a reasonable score for 2-layer models with 256 hidden units."
	"We then used the WMT De!En 2016 test set (2,999 examples) to compute the hallucination percentage for each model."
	I checked the WMT official website http://matrix.statmt.org/matrix. It shows that the best result was a BLEU score of 40.2, which was obtained at 2016. The models used in this work are about 20.0, which are much less than the WMT results reported two years ago. Note that neural machine translation has made remarkable progress in recent two years, not to mention that production systems like Google translator perform much better than research systems. Therefore, the discoveries reported in this work are questionable. I strongly suggest the authors to conduct the studies base on the latest NMT architecture, i.e., Transformer.

	Furthermore, I checked the examples given in  introduction in Google translator and found no hallucination. So I'm not sure whether such hallucinations are really critical to today's NMT systems. I'd like to see that the study on some production translation systems, e.g., applying Algo 1 to Google translator and check its outputs, which can better motivate this work.

	For the analysis in Section 6.1, if attention is the root cause of hallucinations, some existing methods should have already address this issue. Can you check whether the model trained by the following work still suffers from hallucinations?
Modeling Coverage for Neural Machine Translation, ACL 16.


We consider the problem of information compression from high dimensional data. Where many studies consider the problem of compression by non-invertible transformations, we emphasize the importance of invertible compression. We introduce new class of likelihood-based Auto-Encoders with pseudo bijective architecture, which we call Pseudo Invertible Encoders. We provide the theoretical explanation of their principles. We evaluate Gaussian Pseudo Invertible Encoder on MNIST, where our model outperforms WAE and VAE in sharpness of the generated images.

PIE extend NICE and Real NVP into situations which require having a smaller dimensionality of the latent variable (d) compared to the dimensionality of the observed variable (D), i.e. d < D. This is done by learning an extension function g(z) from R^d to R^{D-d} and then using the change of variables formula on x and [z, g(z)]. To model probabilistically the deterministic function g(z) is replaced by Normal distribution with mean g(z) and a small variance.
PIE is used to build deep generative models and trained on the MNIST dataset. The authors show that the models learnt via PIE produce sharper samples than VAEs and Wasserstein autoencoders (WAEs). No comparison to real NVP is made, which should be the main baseline of comparison to answer the question of "what is the advantage of having d < D?". Further MNIST is no longer a good enough benchmark to evaluate deep generative models. Most representative work in this literature use CIFAR-10, downsampled Imagenet, or Imagenet at 256x256.
This work falls short of the standards of ICLR in a few ways:
1. The presentation is unclear. The explanation of the extension-restriction idea is overly complicated. Further, the paper does very little to properly contextualize this work in the literature. Real NVP and flow-based models are mentioned but the proposed technique is not compared to it. The authors say they "introduce new class of likelihood-based Auto-Encoders", but this is false as far as I understood. The technique is not even an autoencoder since a separate decoder is not trained, and is obtained by exactly inverting the encoder as in real NVP.
2. The experiments are weak. The samples shown are of poor quality, and on a very simplisitic dataset (MNIST). The authors compare with vanilla VAEs, but ignore more recent improvements to VAE such as VAE-IAF, flow-based models, and also autoregressive models. A heuristic is used to measure sharpness and only used to compare against VAE and WAE. Since all these models allow likelihood evaluation, likelihoods should also have been compared.
3. The technique itself is a small change over real NVP and it's not clear whether this change brings any improvements or provides any insights about generative modeling.


In contrast to the standard classification paradigm where the true (or possibly noisy) class is given to each training pattern, complementary-label learning only uses training patterns each equipped with a complementary label. This only specifies one of the classes that the pattern does not belong to. The seminal paper on complementary-label learning proposed an unbiased estimator of the classification risk that can be computed only from complementarily labeled data. However, it required a restrictive condition on the loss functions, making it impossible to use popular losses such as the softmax cross-entropy loss. Recently, another formulation with the softmax cross-entropy loss was proposed with consistency guarantee. However, this formulation does not explicitly involve a risk estimator. Thus model/hyper-parameter selection is not possible by cross-validationwe may need additional ordinarily labeled data for validation purposes, which is not available in the current setup. In this paper, we give a novel general framework of complementary-label learning, and derive an unbiased risk estimator for arbitrary losses and models. We further improve the risk estimator by non-negative correction and demonstrate its superiority through experiments.

This paper proposes an improved approach to the "complementary-label" form of weak supervision, in which a label that is *not* the true label is marked. Specifically, this paper proposes an unbiased estimator that accepts arbitrary loss functions and models. Noting that this proposed estimator can suffer from overfitting due to unbounded negative loss, a lower-bounded estimator is proposed. Experiments are then performed on several image classification datasets.
Pros:
- This paper addresses a creative form of weak supervision, proposed by prior work, in which labels that are *not* the true label are labeled, in a clear fashion.
- The first proposed estimator is unbiased, as shown by a proof, and accepts arbitrary losses, an improvement over prior approaches
- The overall presentation is clear and clean
Cons:
- One of the main claims of the paper is the proposal of an unbiased estimator. However, this estimator then does not seem to work well enough due to degenerate negative loss.  So then a modified version is proposed- which does not appear to be unbiased?  Either way, no assertion or proof of it being unbiased is given.  So then presumably this also reverses the claim of being able to cross-validate?  This seems like a major weakening of the paper's contributions
- Since the unbiased estimator does not appear to work well, two implementations of a corrected one are proposed, using heuristic approaches without explicit theoretical guarantees.  This shifts the burden to the experimental studies.  These are somewhat thorough, but not extremely so: for example, one set of hyperparameters were used for all of the methods?  This seems like it could implicitly handicap / favor some over others?
- The proposed estimator is based on the assumption that the probability of classes in the complement set (the set of labels other than the one marked as incorrect) is uniformly distributed (e.g. see beginning of Proof of thm 1).  However, this seems like a potentially naive assumption. Indeed, in the related work section, it is mentioned that work in 2018 already considered the case where this uniformity assumption does not hold.
- More broadly, but following from the above: The paper does not provide any real world examples, real or hypothetical, to give the reader an idea of whether the above uniformity assumption---or really any of these assumptions---are well-motivated or empirically justified.  At the bottom of page 3 in the related work, a concrete application used in prior work is mentioned---where crowd workers are shown single labels and vote Y/N, leading to a mix of standard (if Y) and complement-labeled (if N) data---however this mixed setting is not considered explicitly in this paper.  So, how is the reader supposed to get any idea of whether the assumed setup is motivated or justified?  The experiments do not provide this, because the complementary labels are synthetically generated according to the model assumed in the paper.  Additionally, it is briefly mentioned that collecting complementary labeled data is faster, but again no concrete examples are given to support this.


Learning domain-invariant representation is a dominant approach for domain generalization, where we need to build a classifier that is robust toward domain shifts induced by change of users, acoustic or lighting conditions, etc. However, prior domain-invariance-based methods overlooked the underlying dependency of classes (target variable) on source domains during optimization, which causes the trade-off between classification accuracy and domain-invariance, and often interferes with the domain generalization performance. This study first provides the notion of domain generalization under domain-class dependency and elaborates on the importance of considering the dependency by expanding the analysis of Xie et al. (2017). We then propose a method, invariant feature learning under optimal classifier constrains (IFLOC), which explicitly considers the dependency and maintains accuracy while improving domain-invariance. Specifically, the proposed method regularizes the representation so that it has as much domain information as the class labels, unlike prior methods that remove all domain information. Empirical validations show the superior performance of IFLOC to baseline methods, supporting the importance of the domain-class dependency in domain generalization and the efficacy of the proposed method for overcoming the issue.

In this paper, the author(s) propose a method, invariant feature learning under optimal classifier constrains (IFLOC), which maintains accuracy while improving domain-invariance. Here is a list of suggestions that will help the author(s) to improve this paper.
1.The paper explains the necessity and effectiveness of the method from the theoretical and experimental aspects, but the paper does not support the innovation point enough, and the explanation is too simple.
2.In this paper, Figure3-(b) shows that the classification accuracy of IFLOC-abl method decreases a lot when γ is taken to 0. Figure3-(c) shows that the domain invariance of IFLOC-abl method becomes significantly worse when γ is 10. The author(s) should explain the reasons in detail.
3. The lack of analysis on domain-class dependency of each dataset makes the analysis of experimental results weak.


We propose Power Slow Feature Analysis, a gradient-based method to extract temporally slow features from a high-dimensional input stream that varies on a faster time-scale, as a variant of Slow Feature Analysis (SFA). While displaying performance comparable to hierarchical extensions to the SFA algorithm, such as Hierarchical Slow Feature Analysis, for a small number of output-features, our algorithm allows fully differentiable end-to-end training of arbitrary differentiable approximators (e.g., deep neural networks). We provide experimental evidence that PowerSFA is able to extract meaningful and informative low-dimensional features in the case of (a) synthetic low-dimensional data, (b) visual data, and also for (c) a general dataset for which symmetric non-temporal relations between points can be defined.

Summary
The manuscript proposes to use power iterations in an approximate "whitening layer" to optimize the slowness objective of SFA in a very general setting. A set of experiments illustrates that this way of doing nonlinear SFA is meaningful.
Quality
Although the idea is pretty straight forward and the paper shows qualitative results on a number of datasets, the relative merit of the approach is empirically not well characterized.
Clarity
The manuscript is in general well written and the technical content is well accessible. However the description of the whitening layer implementation needs some more details.
Originality
The idea of using a whitening layer together with the slowness objective has not been explored before. There is a second ICLR 2019 submission (Pfau et al.) with a very similar idea, though.
Empirical Evaluation
The approximate whitening should lead to a trade-off between whitening and slowness optimization. I miss an experiment illustrating that trade-off. Also the comparison to nonlinear SFA using expansion or kernelization of hierarchical SFA is empirically not properly characterized. In the end, if one takes the slowness objective seriously, one would use the method yielding slower results.
Significance
The manuscript introduces a way of running nonlinear SFA with approximate constraints in a general deep learning setting with a differentiable implementation using a dedicated whitening layer based on power iterations.
Reproducibility
The data is either synthetic or publicly available. The Keras implementation of the PowerWhitening layer as well as the entire neural network along with its optimization schedule is not shared. Hence, there should be some effort involved to reproduce the experiments.
Pros and Cons
1+) The idea of an approximate whitening layer is conceptually simple and clear.
2-) The description of the practical implementation of the power iteration is slightly imprecise.
3-) The algorithm scales badly in the number of output dimensions. This scaling is bad in a computational sense and also in a statistical sense.
Details
a) Section 6.1: Why do you need to add the noise term? What is the statistical meaning of this added noise?
b) Section 6.1: the solutions if comparable -> the solutions is comparable
c) References: Shaham -> ICLR 2018 paper
d) References: nyström -> Nyström
e) The name for the algorithm "Power SFA" is a little bit bold.


Semi-supervised learning (SSL) is a study that efficiently exploits a large amount of unlabeled data to improve performance in conditions of limited labeled data. Most of the conventional SSL methods assume that the classes of unlabeled data are included in the set of classes of labeled data. In addition, these methods do not sort out useless unlabeled samples and use all the unlabeled data for learning, which is not suitable for realistic situations. In this paper, we propose an SSL method called selective self-training (SST), which selectively decides whether to include each unlabeled sample in the training process. It is also designed to be applied to a more real situation where classes of unlabeled data are different from the ones of the labeled data. For the conventional SSL problems which deal with data where both the labeled and unlabeled samples share the same class categories, the proposed method not only performs comparable to other conventional SSL algorithms but also can be combined with other SSL algorithms. While the conventional methods cannot be applied to the new SSL problems where the separated data do not share the classes, our method does not show any performance degradation even if the classes of unlabeled data are different from those of the labeled data.

This paper describes the method for performing self-training where the unlabeled datapoints are iteratively added to the training set only if their predictions by the classifier are confident enough. The contributions of this paper are to add datapoints based on the prediction of the confidence level by a separate selection network and a number of heuristics applied for better selection. On the experimental side, the contribution is to test the scenario where datapoints from irrelevant classes are included in the unlabeled dataset.
The paper is written in a way that makes following it a bit difficult, for example, the experimental setups. Also, the writing can be improved by making the writing more concise and formal (examples of informal: "spoil the network", "model is spoiled", "problem of increased classes", "many recent researches have been conducted", "lots of things to consider for training", "supervised learning was trained" etc.). The contributions of the method could also be underlined more clearly in the abstract and introduction. The description of consistency regularisation methods in section 2.2 is not very clear and I would like to get better understanding of temporal ensembling and SNTG methods here as they play an important role in the experiments.
The idea of selective sampling for self-training is promising and the investigated questions are interesting. As far as I understand, the main contribution of this paper is the use of separate "selection network" to estimate the confidence of predictions by "classification network". However, as the "selection network" uses exactly the same input as "classification network", it is hard to imagine how it can learn additional information. For example, imagine the case of binary classification. If the selection network predicts 0 in come cases, it can be used to improve the result of "classification network" by flipping the corresponding label. How can you interpret such a thought experiment? One could understand the use of "selection network" as a way to automatically select a threshold of what to consider confident, however, in this case, the prediction of "selection network" should be thresholded at 0.5 (correct prediction or not), but the experiments use complex thresholds. Could you elaborate more on why the selection network is needed? How would it compare to a simple strategy of only including the datapoints whose top-1 prediction of "classification network" is greater than some threshold? Finally, could you show a plot of top-1 prediction of "classification network" vs score of "selection network" and elaborate on that?
Then, in sections 3.2 and 3.3 the authors introduce a few additional tricks for self-training: exclude datapoints whose predictions are changing and balance the classes. Intuitively, these criteria are well motivated, but unfortunately, the combination of all the intuitions (including "selection network" with threshold) is not very principled. Ablation study shows that the use of the "selection network" strategy does not improve the results without these heuristics. It would be interesting to see how these heuristics would do without "selection network", for example, either by doing simple self-training with thresholding on the score of the classifier or by applying only these heuristics in combination with TempEns+SNTG. In the current form of evaluation, it is hard to say if there is any benefit of using the "selection network" that is the main novelty of the paper.
It is very valuable that the experimental results include many recently proposed methods. Besides, the settings are described in details that could help for the reproducibility of the results. However, I have a few concerns about the results. First of all, the proposed SST algorithm alone only performs better than baselines in 1 case, equal to them in 1 case and worse in 1 (table 3). Besides, as the base classifier is different for various baselines, it is hard to compare the methods. Then, the important hyperparameter of the method---threshold---seems to be hard to select (both in sections 4.1 and 4.2). How did you chose the current values? How sensitive is it? Why various datasets need different settings? How the threshold value can be set in practice? Another important parameters is the number of iterations of the algorithm. How was it chosen? Concerning the experiments of section 4.2, how would the baseline methods of section 4.1 do in this case? Why did you select to study animal vs non-animals sets of classes? What would happen if you use random class splits or split animal classes (like in a more realistic scenario)?
To conclude, while I find the studied problem quite interesting and intuitions behind the method very reasonable, the current methodology is not very principled and the experiment evaluation did not convince me that such an elaborate strategy is needed.
Some questions and comments:
- The setting of including unrelated classes in the unlabeled data resembles transfer learning setting. Could you explain why the ideas from transfer learning are not applicable in your case?
- In the training procedure of "selection network" of Sections 3.1, do you use the same datapoints to train a "classification network" and "selection network"? If it is the case, how do you insure that the "classification network" does not learn to fit the data perfectly and thus all labels s_i are 1?
- In the last sentences of the first paragraph on p.2 you make a contrast between using softmax and sigmoid functions, however, normally the difference between them is their use in binary or multiclass classification. Is there anything special that you want to show in you case?
- What do you mean in section 3.3 by "if one class dominates the dataset, the model tends to overfit"?
- I think parameters of training the networks from the beginning of section 4 could be moved to the supplementary materials.
- Figure 3: wouldn't the plot of accuracy vs amount of data be more suitable here?
- Synthetic experiments of supplementary materials: the gains of the methods seem to be small. What are the numerical results? What would happen if you allow to select starting point at random (a more realistic case)?
- Can you explain the sentence "To prevent data being added suddenly, no data was added until 5 iterations"?
- How was it possible to improve the performance in experiment of section 4.2 with 100% of irrelevant classes?