Soft-Gated Warping-GAN for Pose-Guided Person Image Synthesis
Despite remarkable advances in image synthesis research, existing works often fail in manipulating images under the context of large geometric transformations. Synthesizing person images conditioned on arbitrary poses is one of the most representative examples where the generation quality largely relies on the capability of identifying and modeling arbitrary transformations on different body parts. Current generative models are often built on local convolutions and overlook the key challenges (e.g. heavy occlusions, different views or dramatic appearance changes) when distinct geometric changes happen for each part, caused by arbitrary pose manipulations. This paper aims to resolve these challenges induced by geometric variability and spatial displacements via a new Soft-Gated Warping Generative Adversarial Network (Warping-GAN), which is composed of two stages: 1) it first synthesizes a target part segmentation map given a target pose, which depicts the region-level spatial layouts for guiding image synthesis with higher-level structure constraints; 2) the Warping-GAN equipped with a soft-gated warping-block learns feature-level mapping to render textures from the original image into the generated segmentation map. Warping-GAN is capable of controlling different transformation degrees given distinct target poses. Moreover, the proposed warping-block is lightweight and flexible enough to be injected into any networks. Human perceptual studies and quantitative evaluations demonstrate the superiority of our Warping-GAN that significantly outperforms all existing methods on two large datasets.

The authors propose a computer vision framework for performing pose transfer, i.e. generating images of a given person in a random pose. The key technical contributions of this approach are 1) exploiting an external human parser for an intermediate step of predicting target segmentations given the input and the target pose, 2) a warping module estimating transformations between the input and the predicted target human parsings.   In my opinion, the paper is rather interesting and original: I like the idea of explicitly learned transformations and its adaptation for the task. There have been a number of works around modeling deformations (for example, [23] estimated local affined transformations for body parts), but the proposed implementation went further by learning transformation fields densely and end-to-end. The experimental results show significant improvements over the state-of-the-art given the chosen standard benchmarks and the metrics.  The manuscript is rather easy to follow and understand, but it still contains a significant number of typos and minor grammar mistakes. Some implementation details are missing which may make this work difficult to reproduce. It would be helpful if the authors provided a more detailed description of the geometric matcher.   My high level concern, however, is that there has been a large number of similar works optimizing for performance (overfitting?) on DeepFashion/Market-1501 datasets, which have rather low variety and cannot be considered representative of real data. I personally would be excited to see us as a community move beyond these benchmarks and target more ambitious settings.   My more detailed comments and questions are below.  The human parser has been trained on LIP dataset constructed from COCO images - therefore, I'd like to see some comments on the quality of obtained segmentations (on DeepFashion, and especially on the low resolution Market-1501 dataset) and how it affects the quality of the generations.  Stage 1 & Figure 2. Does the input to the trained parser include the original RGB image? L130-131 suggest so, while Figure 2 doesn't. Are heatmaps and probability maps concatenated? How is the background label treated?  Figure 3. Is the Geometric Matcher module trained separately or jointly with the rest of the generator? What is the meaning of the block on the right of the Matching layer? Adding labels to this figure would be helpful.  Geometric Matcher & Warping Block. The abbreviation TPS is used before it's defined.  What makes the network generalize on the body parts that are not observed in the source image (for example, going from front to back)? The description of the warping block would be easier to understand if the authors showed the corresponding notations in Figure 4. Why is learning facilitated by decomposing the result into the original set of features and residuals? How is the network's performance affected if the warping is done directly? The term "soft gate" is unclear in this context and its position in Figure 4 is not obvious.  Discriminator architecture. In the proposed implementation, the discriminator takes an exhaustive set of inputs (input image, target keypoints, source parsing, target parsing, generated image) - have the authors analyzed the importance of each of these components?  Objective functions. It would be easier for the reader if the authors explicitly described their loss terms (L1 or L2 for L_pixel, base network for L_perceptual, the exact kind of GAN objective). Stabilizing discriminators by matching intermediate features at multiple scales has been done before - the authors may want to cite: Wang et al. High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs. CVPR, 2018.  Datasets. How is the parser used to remove failure cases from the dataset, assuming that there is no ground truth for this data?  How is the training / test splits are defined? The number of training / test images in this paper differs from, for example, [23]. Assuming it was arbitrary, did the authors retrain the state-of-the-art models on the same split? Otherwise, the comparison may be unfair (images from the training set used for evaluation).  Symbol beta is overloaded (used in Eq. (2) and also to define Adam's parameters). How are betas in (2) are defined?  Using Inception score has been a common place in similar works, but one should keep in mind that it's not strictly appropriate for the task and in practice is highly unstable (can be high for "blurry" images obtained with L1/L2 loss). I'm not sure what the authors mean saying that high IS scores confirm generalization (L239). Also, it would be useful to report IS for the real images.  Human study: what was the exact protocol of the experiments? (number of images, number of workers, number of experiments / worker, how the images were selected, image resolution). It is somewhat disappointing that in these experiments the workers were asked to evaluate the "realism" of the generations, but not how well the target pose is captured or whether the identity and the appearance of the person from the original image are preserved. It would be interesting to see more results on these aspects as well. Why doesn't Table 2 contain comparison of the proposed method's outputs with ground truth images?   Nitpicking:  L119: we ARE following L136: please reformulate "parsing obey the input pose" L142: number of pixelS L162: Obtained -> Having obtained? L197: please rephrase "large arrange among of poses" L228: Figure shown -> Figure shows L246: significantLY bettter L268: generator significantly improveS Paper formatting seems to have suffered by reducing the manuscript to 8 pages, please fix.


Learning Plannable Representations with Causal InfoGAN
In recent years, deep generative models have been shown to 'imagine' convincing high-dimensional observations such as images, audio, and even video, learning directly from raw data. In this work, we ask how to imagine goal-directed visual plans -a plausible sequence of observations that transition a dynamical system from its current configuration to a desired goal state, which can later be used as a reference trajectory for control. We focus on systems with high-dimensional observations, such as images, and propose an approach that naturally combines representation learning and planning. Our framework learns a generative model of sequential observations, where the generative process is induced by a transition in a low-dimensional planning model, and an additional noise. By maximizing the mutual information between the generated observations and the transition in the planning model, we obtain a low-dimensional representation that best explains the causal nature of the data. We structure the planning model to be compatible with efficient planning algorithms, and we propose several such models based on either discrete or continuous states. Finally, to generate a visual plan, we project the current and goal observations onto their respective states in the planning model, plan a trajectory, and then use the generative model to transform the trajectory to a sequence of observations. We demonstrate our method on imagining plausible visual plans of rope manipulation 3 .

1. Summary - This paper proposes Causal InfoGAN which is capable of generating a realistic sequence of observations from a given initial observation to a goal observation. The motivation is that this is solving a relaxed version of planning problem, which requires generating a sequence of actions. The idea of Causal InfoGAN is to train a generator that generates two consecutive observations given two consecutive “abstract states”, which encourages the model to generate realistic transitions. In order to enforce the consistency between abstract states and observations, this paper adds a mutual information maximization objective (which is the idea from InfoGAN). The experimental result on a toy 2D domain shows that the proposed model can learn reasonable abstract states that are consistent with the dynamics of the environment, as opposed to simple baselines (e.g., K-means) which are independent of the dynamics of the environment. The qualitative result on Rope Manipulation data shows that Causal InfoGAN can generate more realistic transitions than InfoGAN.   [Pros] - Novel application of GAN that is capable of generating a sequence of observations that is consistent with the dynamics of the environment. - The proposed idea is sensible and interesting.   [Cons] - The experimental result is okay but not much comprehensive.  2. Quality - The idea of training a GAN to generate realistic transitions that is interesting.  - The experimental result could be more comprehensive. First, the paper only presents qualitative results for Rope Manipulation task. Though Casual InfoGAN looks better than InfoGAN, some transitions seem very unrealistic. It would be better to provide quantitative results (something like Table 1). Second, it would be much convincing/interesting to show results on more complex and popular domains such as Atari and MuJoCo. Lastly, it would be good to show an evidence that the learned model can generalize to unseen observations, which would be one of the advantages of model-based RL. - It would be good to discuss the following work in the related work section: Universal planning network [Srinivas et al.] and Model-Based Planning with Discrete and Continuous Actions [Henaff et al.]. They seek to generate a sequence of “actions” given initial/goal observations.  3. Clarity - The problem formulation and the description of the method are well-written. - In Section 2, the paper claims that the planning problem (generating a sequence of actions) is “unnecessarily” difficult to solve, in order to justify their relaxed problem (generating observations). This sounds like a too strong argument. How can we build a control agent if we do not care about generating actions and only generate “observations”?   4. Originality - The idea of learning an abstract transition model using GAN is novel to my knowledge.  5. Significance - I would personally view this work as an interesting application of GAN for dynamical systems rather than “learning plannable representation” that sounds quite ambitious, because the authors are not tackling the full planning problem (e.g., dealing with actions, exploration, reward). Nevertheless, I think this paper is tackling an interesting problem with a new idea, which can be potentially useful for control in the future.


Meta-Gradient Reinforcement Learning
The goal of reinforcement learning algorithms is to estimate and/or optimise the value function. However, unlike supervised learning, no teacher or oracle is available to provide the true value function. Instead, the majority of reinforcement learning algorithms estimate and/or optimise a proxy for the value function. This proxy is typically based on a sampled and bootstrapped approximation to the true value function, known as a return. The particular choice of return is one of the chief components determining the nature of the algorithm: the rate at which future rewards are discounted; when and how values should be bootstrapped; or even the nature of the rewards themselves. It is well-known that these decisions are crucial to the overall success of RL algorithms. We discuss a gradient-based meta-learning algorithm that is able to adapt the nature of the return, online, whilst interacting and learning from the environment. When applied to 57 games on the Atari 2600 environment over 200 million frames, our algorithm achieved a new state-of-the-art performance.
The central goal of reinforcement learning (RL) is to optimise the agent's return (cumulative reward); this is typically achieved by a combination of prediction and control. The prediction subtask is to estimate the value function -the expected return from any given state. Ideally, this would be achieved by updating an approximate value function towards the true value function. The control subtask is to optimise the agent's policy for selecting actions, so as to maximise the value function. Ideally, the policy would simply be updated in the direction that increases the true value function. However, the true value function is unknown and therefore, for both prediction and control, a sampled return is instead used as a proxy. A large family of RL algorithms [Sutton, 1988, Rummery and Niranjan, 1994, van Seijen et al., 2009, Sutton and Barto, 2018, including several state-of-the-art deep RL algorithms [Mnih et al., 2015, van Hasselt et al., 2016, Hessel et al., 2018, Espeholt et al., 2018, are characterised by different choices of the return.
The discount factor γ determines the time-scale of the return. A discount factor close to γ = 1 provides a long-sighted goal that accumulates rewards far into the future, while a discount factor close to γ = 0 provides a short-sighted goal that prioritises short-term rewards. Even in problems where long-sightedness is clearly desired, it is frequently observed that discounts γ < 1 achieve better results [Prokhorov and Wunsch, 1997], especially during early learning. It is known that many algorithms converge faster with lower discounts [Bertsekas and Tsitsiklis, 1996], but of course too low a discount can lead to highly sub-optimal policies that are too myopic. In practice it can be better to first optimise for a myopic horizon, e.g., with γ = 0 at first, and then to repeatedly increase the discount only after learning is somewhat successful [Prokhorov and Wunsch, 1997].
The return may also be bootstrapped at different time horizons. An n-step return accumulates rewards over n time-steps and then adds the value function at the nth time-step. The λ-return [Sutton, 1988, Sutton andBarto, 2018] is a geometrically weighted combination of n-step returns. In either case, the meta-parameter n or λ can be important to the performance of the algorithm, trading off bias and variance. Many researchers have sought to automate the selection of these parameters [Kearns and  Singh,
There are potentially many other design choices that may be represented in the return, including off-policy corrections [Espeholt et al., 2018, target networks [Mnih et al., 2015], emphasis on certain states [Sutton et al., 2016], reward clipping [Mnih et al., 2013], or even the nature of the rewards themselves [Randløv and Alstrøm, 1998, Singh et al., 2005, Zheng et al., 2018.
In this work, we are interested in one of the fundamental problems in reinforcement learning: what would be the best form of return for the agent to maximise? Specifically, we propose to learn the return function by treating it as a parametric function with tunable meta-parameters η, for instance including the discount factor γ, or the bootstrapping parameter λ [Sutton, 1988]. The meta-parameters η are adjusted online during the agent's interaction with the environment, allowing the return to both adapt to the specific problem, and also to dynamically adapt over time to the changing context of learning. We derive a practical gradient-based meta-learning algorithm and show that this can significantly improve performance on large-scale deep reinforcement learning applications.

A gradient-based hyper-parameter technique for reinforcement learning is proposed, and implemented on top of the IMPALA architecture. The algorithm consists in estimating the gradient of an objective J’ w.r.t. hyper-parameters eta, using the chain rule: d_J’/d_eta = d_J’ / d_theta’ * d_theta’ / d_eta, where theta’ = theta + f(tau, theta, eta) is the updated vector of model parameters after observing trajectory tau with current parameters theta and hyper-parameters eta. In other words, we look for a small change in hyper-parameters that would improve the performance of the agent after a parameter update, as measured by the objective J’. This objective is based on independently sampled trajectories tau’ and a fixed set of hyper-parameters eta’. In order to accommodate the fluctuations in hyper-parameters eta, they are provided as input to the actor and critic networks. The main experiments are on Atari, where eta = {gamma, lambda} (respectively the discount factor and the bootstrapping parameter in TD(lambda)). They show a significant improvement over the standard IMPALA architecture (which uses the best gamma = 0.995 and lambda = 1 found by grid search), in terms of median human-normalized score.  There is no doubt this is an interesting paper. Just the fact that the discount factor matters so much (huge improvement with vanilla IMPALA when going from 0.99 to 0.995, but poor results at 0.999) is quite notable to me: to my knowledge it is not something that had been investigated previously (on Atari specifically). I actually find this somewhat worrying, since it sheds doubts on the validity of the standard practice of training with gamma = 0.99 then evaluating on the total accumulated reward for benchmarking purpose, as is done in almost all deep RL papers evaluated on Atari... but this is another discussion.  The core empirical results are convincing enough: there is a relatively consistent improvement over the baseline IMPALA, which is a solid benchmark. It is still pretty far from Ape-X DQN but since the number of environment steps is much lower, this may not be a fair comparison to make. As shown in Fig. 2 of the Appendix, though, there are some games where the proposed technique clearly hurts: it would have been interesting to try and understand what is going on in these games that makes the hyper-parameter adaptation scheme fail. In addition, I believe there should have been some comparison to other hyper-parameter adaptation techniques, since the ability to improve on a fixed gamma over all games is not particularly surprising (given the highly varying nature of Atari games). At the very least, a comparison to grid search should have been done where one would pick the best gamma *for each game*.  I would also argue that the proposed method is not just a hyper-parameter adaptation scheme: since hyper-parameters eta are provided as input to the actor and critic, the model is essentially performing multi-task learning (learning to predict returns at multiple timescales). It has been observed before that additional tasks can have a beneficial effect (ex: the UNREAL agent), and in my opinion this begs the question as to whether the improvement comes mostly from changing eta or from learning to predict over multiple eta’s. Note that even if eta eventually converges to a fixed value, the optimization w.r.t. J’ encourages updates that give good results for eta’ != eta, so the model is still learning with at least two time scales. Related to this last point, the motivation given by the authors for adding eta as a model input is that “the return function gη(τ ) is non-stationary”, but it is not clear to me whether this is the main one, since eta does not change that fast. Another clearly important factor, not mentioned in the paper, is that it allows to compute J’ on eta’ != eta, while otherwise it would seem difficult to optimize J’ with the “wrong” hyper-parameters (by the way please mention 1.4 earlier, or add a reference to it in 1.2, because eq. 11 got me puzzled for a while due not being aware of that yet). To summarize, my point is that even if there is no doubt the method works, I wish there had been more efforts to understand why.  In this respect, the illustrative examples from Section 2 could have help shed light on the inner workings of the algorithm. Unfortunately I found them extremely confusing. I suspect that my understanding of “state-dependent” hyper-parameter may not match the actual implementation. For instance when I think of a state-dependent gamma, to me it means that the Bellman equation would be written V(s) = E[R(s)] + gamma_s V(s’), where gamma_s is the state-dependent discount factor. But then it makes no sense for gamma_s to be near zero in “noise” states in Fig. 1a, as it would mean we only learn V(s) from the noise R(s), which can’t be right. The algorithm also learned a specific high value for gamma_9 while gamma should not matter in that state, since the next state is terminal and thus V(s’) = 0. The second example (Fig. 1b) is equally confusing, if not more. Since even-numbered states are aliased, the agent sees them as a single state. That state is visited as often as a bottleneck state (contradicting “The fully observed bottleneck states are visited frequently”), and has essentially the same properties regarding the noise in the (immediate) reward distribution: a random (due to aliasing) number chosen among -r1, .. -rn (vs. among r1, ..., rn in a bottleneck state). I may have misunderstood the meaning of “aliased” states, in which case a clarification would be appreciated. In any case, I was unable to derive a better understanding of the proposed method from these two toy examples.  There are other unclear parts in the paper, even if the high-level idea is easy to grasp. One of them is how the dependence of the actor pi and critic v on the hyper-parameters eta is handled w.r.t. gradient computations: the derivations in 1.2 and 1.3 assume that v and pi do not depend on eta. But then 1.4 adds such a dependency... Are you actually backpropagating through the whole computational graph, or “cutting” this specific part of the gradient? (and in the former case, does that include backpropagating through the IMPALA importance sampling weights?). I believe this is an important point to clarify and motivate. Other unclear points include: - The inclusion of d_thera / d_eta in eq. 3 (first term). I do not understand what it is supposed to represent. From my point of view, when starting from theta, modifying eta has no effect on theta itself: it can only change future values theta’. The “accumulative trace” approximation looks fishy to me as I fail to see what it is meant to approximate: I see it simply as a momentum applied to d_f / d_eta. Note also that when reading the text it is not obvious that this is a recursive update with z’ being the new value of z used in the nest iteration. - The motivation for keeping only the policy gradient objective when defining J’ in eq. 14. It is not obvious to me that this is the best choice, since a better critic v could also have a positive impact on the agent’s performance. - The motivation for using gamma’ = 0.995 in Atari experiments: should not it be chosen based on the performance of the adaptive algorithm, rather than the best fixed gamma? - The exact meaning of batches (“meta batch size” in Table 2): is it the case that J’ is actually evaluated over a mini-batch of trajectories, instead of a single trajectory tau’ as the equations suggest? - More generally, adding the pseudo-code of the algorithm in the Appendix would be very helpful, especially with 1.5 showing there are important implementation “details”  Minor points (missing from original review, but thankfully the character limit is gone): - I would suggest to remove eta from the definition of n-step returns in eq. 6 since the proposed method can’t optimize discrete hyper-parameters (a limitation, by the way, which is not clearly stated in the abstract & conclusion) - In eq. 8 dividing J by two would slightly simplify subsequent equations - Caption of Fig. 1: “the first one shows how the meta-parameter γ or λ adapts” => please specify “for each state” - What is the value of n in experiments? Is it the “unroll length” 20? - l. 177 please refer to Fig. 1 (top), as trying to understand the examples from their text description is difficult - Please specify the values of respectively lambda and gamma used in the two illustrative examples - l. 202 why not write eta = {gamma, lambda} ? - In Atari do you use lambda’ = 1? If yes how do you deal with the fact a sigmoid can’t be equal to 1 (footnote 2) and you need the logit as input to the actor and critic to compute J’? (the same question actually arises for the illustrative examples) - In Appendix Fig. 2 and 3 please specify eta’  Regarding the author response:  First, thank you for the detailed response. My main concerns have been addressed by the extra results related to the grid search on gamma and the multi-task experiment, so I've changed by recommendation to accept. A few follow-up remarks that hopefully you can take into account when preparing the final version:  1. You motivated including d_theta / d_eta in eq. 3 with a recurrent view of theta as a function of eta (response l. 35). However, in practice eta has not been kept fixed during the previous steps (since the algorithm is adapting it), and thus theta is not just a function of the current eta, but depends of the value of eta at each previous timestep. As a result this derivative still makes no sense to me (NB: and even if eta had been kept fixed, I am still not sure it would be a good idea to follow this gradient, since we're never going back in time to "improve the past" -- changing eta only affects future updates)  2. In your first toy example (Fig. 1a) you say that (response l. 27) “in noisy states setting e.g. gamma2 ≈ 0 avoids receiving the "noise" reward”. However this effectively shuts down all subsequent rewards as well when e.g. computing the value function for the very first state. I don’t see how one can properly estimate the long-term value v_1 of the first state (with gamma’ = 1) if gamma_2 = 0 since the only return the agent will see from state 1 is 0.1.  3. Your second example (Fig. 1b) finally "clicked" for me when I wrote down what the return would look like depending on when to bootstrap (with binary lambda's). For instance using as target either r_i + v(aliased_state) or r_i - r_i + v(bottleneck_state). It becomes clear that the second one is less noisy since the two consecutive rewards immediately cancel each other in the return. So maybe you could write it down explicitly in the paper as well (possibly in the Appendix if not enough room).  4. Regarding point R2.7 in your response, I actually didn't think of using gamma'_t = gamma_t. I had in mind a simpler "classical" hyper-parameter selection strategy, e.g. doing a grid-search over gamma'. I don't disagree that in the absence of other any information, using gamma' = the best fixed gamma seems like a sensible solution, however it remains a somewhat arbitrary choice that was not motivated in the submission.  Thanks!  ========  One last thing, regarding point 1: I posted the following on the reviewers’ discussion board, to try and explain my point of view to fellow reviewers (none of them could understand the role of mu by the way, so I really hope this part gets fixed in a final version):  I see the meta-optimization problem as an iterative procedure, where at each step we start from the previous theta and want the update f (as in eq. 1) to improve on the meta-objective J'. We can control f through hyper-parameters eta, and eq. 2 tells us the direction to follow in the hyper-parameter space.  This direction requires computing d_theta' / d_eta, which from eq. 1 is *equal* to d_f / d_eta, because theta is the (fixed) starting point at the current step, and is not affected by any subsequent change to eta. Plugging this in eq. 2, we get that d_J' / d_eta = d_J' / d_theta' * d_f / d_eta, which is indeed the update direction that is being used when when mu = 0 (see eq. 5 where z' = d_f / d_eta).  So, to me, mu=0 is not an approximation, it yields the correct update direction for the optimization objective. Using mu > 0 amounts to adding momentum to the gradient d_f / d_eta, which I guess could be argued to stabilize training in some situations. Note that it's not exactly similar to regular momentum in SGD because this momentum is not applied to the final update direction (eq. 2), but only to one of its components. So it's not really clear what's going to happen with mu > 0, and since all experiments reported in the submission are with mu=0, there's no way to tell how sensitive performance is to changes to this parameter.  That being said, since from my point of view mu=0 is the natural thing to do, I would not reject the paper based only on its lack of clarity regarding mu's role.


Why so gloomy? A Bayesian explanation of human pessimism bias in the multi-armed bandit task
How humans make repeated choices among options with imperfectly known reward outcomes is an important problem in psychology and neuroscience. This is often studied using multi-armed bandits, which is also frequently studied in machine learning. We present data from a human stationary bandit experiment, in which we vary the average abundance and variability of reward availability (mean and variance of the reward rate distribution). Surprisingly, we find subjects significantly underestimate prior mean of reward rates -based on their self-report on their reward expectation of non-chosen arms at the end of a game. Previously, human learning in the bandit task was found to be well captured by a Bayesian ideal learning model, the Dynamic Belief Model (DBM), albeit under an incorrect generative assumption of the temporal structure -humans assume reward rates can change over time even though they are truly fixed. We find that the "pessimism bias" in the bandit task is well captured by the prior mean of DBM when fitted to human choices; but it is poorly captured by the prior mean of the Fixed Belief Model (FBM), an alternative Bayesian model that (correctly) assumes reward rates to be constants. This pessimism bias is also incompletely captured by a simple reinforcement learning model (RL) commonly used in neuroscience and psychology, in terms of fitted initial Q-values. While it seems sub-optimal, and thus mysterious, that humans have an underestimated prior reward expectation, our simulations show that an underestimated prior mean helps to maximize long-term gain, if the observer assumes volatility when reward rates are stable, and utilizes a softmax decision policy instead of the optimal one (obtainable by dynamic programming). This raises the intriguing possibility that the brain underestimates reward rates to compensate for the incorrect non-stationarity assumption in the generative model and a simplified decision policy.

I have updated my score in light of the proposed revisions. I like this paper and would be happy if this gets accepted.  I liked this paper. The authors show that participants underestimate prior reward rates in a bandit task, a behavior which is well-captured by their dynamic belief model, but not by a fixed belief model and only partially by a traditional RL-model. Interestingly, this initial pessimism might actually be adpative under the assumption that participants expect non-stationary environments and map expectatios onto choices by using a softmax-policy.  - I like the experimental paradigm. I would have maybe shown the fishing reports as an icon array with green and red dots just to make it comparable to the actual task. Was there any spatial correlation between participants' mean estimates for different options? It could be that they thought closer options on the lake might produce similar outcomes.  - this paradigm also reminds me of a paradigm recently applied by Gershman, in a paper called "Uncertainty and exploration", in which he marked different options as high or low variance to assess participants' exploration strategies  - I wonder how this relates to other models of human exploration, for example the finding that humans are often well-captchured by a combination between a Kalman filter learning algorithm and a Thompson sampling exploration strategy (for example, Speekenbrink & Konstantinidis, 2015). Thompson sampling is also known to be equivalent to probability matching, a heuristic that has been frequently called suboptimal but can actually perform quite well under the assumption of nonstationary environment. This also relates back to converging evidence that participants might use a combination of direct and random exploration in multi-armed bandit tasks, and I'm not sure how that can be incoporated by the DBM. I guess a direct incorporation of information gain in the model is similar to a directed exploration strategy, whereas the softmax parameter might track (value-based) random exploration.   - would have been nice to ask participants abut their confidence in the estimated reward rate too, just to see if this is more based on uncertainty of the environment or more deliberate  - how were participants rewarded in this task?   - as a side note, Wu et al. (Generalization and exploration in vast spaces) found that participants tend to under-estimate the spatial correlation of rewards and that this "pessimism bias" turned out to be beneficial, even for algorithms of Bayesian optimization; I think this is quite similar to your temporal finding  Minor comments:  - I find the coloring scheme for some of the plots a little confusing, sometimes purple is FBM, sometimes it's human data; I would try and make this consistent across plots - I'm not sure how the error bars were calculated, is this a mean over participant-nested estimates?  - line 79: p < 10 − 3 should be $10^{-3}$ - Figure 3: I would make all of the plots equi-sized; I'm also not sure how the models were run here, just with the overall mean or with a mean per participant? I guess it's the overall mean, but I would perhaps say that - References: make sure that journal names are capitalized appropriately, for example "computational biology" should be "Computational Biology"


DVAE#: Discrete Variational Autoencoders with Relaxed Boltzmann Priors
Boltzmann machines are powerful distributions that have been shown to be an effective prior over binary latent variables in variational autoencoders (VAEs). However, previous methods for training discrete VAEs have used the evidence lower bound and not the tighter importance-weighted bound. We propose two approaches for relaxing Boltzmann machines to continuous distributions that permit training with importance-weighted bounds. These relaxations are based on generalized overlapping transformations and the Gaussian integral trick. Experiments on the MNIST and OMNIGLOT datasets show that these relaxations outperform previous discrete VAEs with Boltzmann priors. An implementation which reproduces these results is available at https://github.com/QuadrantAI/dvae.

# Response to author feedback  Having read the authors' response I am happy with my original assessment that this would be a good addition to the conference and vote for including this submission. The authors' discussion of the potential reasons for the relatively poor performance of the Gaussian integral trick method in their feedback in response to a comment in my review seem reasonable and from my reading they also adequately address the points raised by the other reviewers.  ---  # Summary and relation to previous work  This submission builds upon several previous pieces of work proposing methods for including discrete latent variables in variational autoencoder (VAE) models (Kingma and Welling, 2014). A key aspect of these works is retaining the ability to train the models with low-variance reparameterisation trick based gradient estimates of the variational objective by relaxing the discrete latent variables with associated continuous valued variables. Of particular significance to this submission are the discrete VAE (dVAE) (Rolfe, 2016) and dVAE++ (Vahdat et al., 2018) models which use a Boltzmann machine (BM) prior on the discrete latent variables and construct a differentiable proxy variational objective by introducing continuous variables zeta corresponding to relaxations of the discrete variables $z$, with $\zeta$ depending on $z$ via a *smoothing* conditional distribution $r(\zeta | z)$. The generative process in the decoder model is specified such that generated outputs $x$ are conditionally independent of the discrete variables $z$ given the continuous variables $\zeta$.  An issue identified with the (differentiable proxy) variational objective used in both the dVAE and dVAE++ approaches is that it is not amenable to being formulated as an importance-weighted bound, with importance-weighted objectives for continuous VAE models having been found to give significant improvements in training performance (Burda et al., 2015). In this submission the authors suggest an alternative dVAE formulation they term dVAE# which is able to use an importance weighted objective. This requires evaluating (/estimating) the log marginal density on the $\zeta$ variables. In general evaluation of the marginal density on $\zeta$ is intractable as it requires marginalising $z$ from joint density on $z$ and $\zeta$, with the exhaustive summation over all $z$ configurations in general intractable to compute.   The authors main contribution is to propose two alternative approaches to overcoming this issue. In the first the smoothing conditional density $r(\zeta | z)$ is assumed to take the same form  as the 'overlapping smoothing transformations' previously proposed in the dVAE++ paper. In this case as the marginalisation over $z$ is intractable, the authors propose fitting a mean field approximation to the BM prior on $z$ and using this factorial approximation to approximate the marginalisation over $z$ to compute approximations to the log marginal density on $\zeta$ and its gradient. The second suggestion is to use a particularly tractable form of continuous relaxation proposed previously for sampling from BM models (Zhang et al., 2012), termed the  'Gaussian integral trick', which allows exact marginalisation over $z$, at the expense over a more complex geometry resulting marginal distribution on the $\zeta$ variables (and so difficulties in choosing an appropriate variational approximation).  The authors also propose generalisations of the 'overlapping smoothing transformations' previously employed in the dVAE++ paper exploit an idea for applying the reparameterization trick to mixture distributions (Graves, 2016) to allow more general forms for the transformations while still remaining reparameterizable, with in particular several transformations which it is suggested will give lower variance gradient estimates being proposed. The proposed dVAE# approach is empirically validated to generally give significant improvements over the previous dVAE and dVAE++ approaches in terms of test set log likelihoods on two benchmark binarized image generative model datasets (MNIST and OMNIGLOT) across a range of different model architectures for each method. The benefit of using an importance weighted objective is validated by a generally increased performance as more importance samples are used and the mean-field approach using a particular 'power-function transformation' instance of their generalised smoothing transformation found to generally give better performance than the other suggested alternatives.  # Evaluation  This submission mainly proposes fairly incremental extensions to the existing dVAE and dVAE++ approaches, however the suggested improvements are motivated well and explained clearly, and the extensive experiments appear to show consistent significant test-set log-likelihood gains for a particular instance of the proposed approach (power-function transform with mean-field approximation) compared to competing dVAE and dVAE++ approaches on a wide range of architectures and across two different datasets. Further the inclusion of the negative results for their being significant gains to using what would seem a perhaps natural alternative of a Gaussian integral trick relaxation, given the ability to sidestep the need to introduce a mean-field approximation in this case, is also an important and interesting empirical contribution. It would be helpful however to have more discussion of potential reasons why the Gaussian integral approach seems to generally not perform as well. Overall though the originality of the proposal is not massive, the quality of the communication of the ideas present and relatively strong empirical validation of the proposals mean I feel the submission would be a worthwhile inclusion in the conference.  # Minor comments / typos / suggestions:  Equation 5: How to get from the second to third line is not necessarily immediately obvious - might be worth noting this relies (I think) on the linearity of the gradient of the augmented energy function with respect to z.  L154: 'covariance matrix becomes diagonally dominant' -> I think this should be precision not covariance matrix?  Digit grouping: in a few places commas are used for thousands digit grouping - generally a thin space should be preferred as a comma can be ambiguous as to whether a decimal point is intended in some locales.  Figures 1 and 2: plots difficult to read at print size due to light weight of text / lines - it would help to plot with fonts, line weights etc. specifically set for small figure size rather than what looks like downscaling a larger plot.  References: several references e.g. [3], [4]. [8], [9], [21] list arXiv pre-print versions when published versions are available. More of a nitpick, but it would also be nicer if all references consistently used initials or full first names. Hamiltonian in reference [41] should be capitalised.


Structure-Aware Convolutional Neural Networks
Convolutional neural networks (CNNs) are inherently subject to invariable filters that can only aggregate local inputs with the same topological structures. It causes that CNNs are allowed to manage data with Euclidean or grid-like structures (e.g., images), not ones with non-Euclidean or graph structures (e.g., traffic networks). To broaden the reach of CNNs, we develop structure-aware convolution to eliminate the invariance, yielding a unified mechanism of dealing with both Euclidean and non-Euclidean structured data. Technically, filters in the structure-aware convolution are generalized to univariate functions, which are capable of aggregating local inputs with diverse topological structures. Since infinite parameters are required to determine a univariate function, we parameterize these filters with numbered learnable parameters in the context of the function approximation theory. By replacing the classical convolution in CNNs with the structure-aware convolution, Structure-Aware Convolutional Neural Networks (SACNNs) are readily established. Extensive experiments on eleven datasets strongly evidence that SACNNs outperform current models on various machine learning tasks, including image classification and clustering, text categorization, skeleton-based action recognition, molecular activity detection, and taxi flow prediction.

The paper proposes a structure-aware convolution structure that can naturally aggregate local inputs from diverse topological structures without predefining data structures. The learnable filters in the convolution operation are well designed in a simple way. The methods shows better quantitative results on eleven datasets when comparing to baselines.  Strength: 1. Simple. The convolution operator is simply unified on diverse topological structures. 2. Extensive experiments on six Euclidean and five non-Euclidean structured datasets.  3. Results show the proposed method has certain improvements over recent convolution neural network and graph convolution networks.  4. The paper is well written, and clear to my understanding.   Weakness: 1. Notation confusion. In equation (1) and theorem #1, x_i denotes a local input from (i-m)-th vertex to (i+m)-th vertex. However, in equation (6),  a same x_i represents a single value (with multi-channel) at i-th vertex only. 2. The detailed information for graph construction is missing, e.g. the way to define edges and construct a graph for non-euclidean datasets. 3. Despite the proposed method shares a different idea with [17], the P_i matrix in theorem #1 mathematically transforms the input x_i into a high-dimensional space where a convolution weights v are applied. Can the authors briefly compares the proposed method with [17]? My feeling is that [17] should be a good baseline to compare on Euclidean datasets.  4. It seems that the baselines in Experiment 5.2 are not the original models, but the authors' own implementations, e.g. ActCNNs[18] has 93% accuracy on Cifar10 instead of 90%. Can the authors comment about this?  5. It would be good if the author can compare number of parameters between the proposed network and used baselines.   Minor comments: 1. In figure 3(f), the improvement looks saturated when using 56 layers network. Will the proposed method also benefit deeper network (e.g. resnet110 in Cifar10)?  2. It would be interesting if the authors would comment on how their approach can be applied to point cloud dataset.  Conclusion: I hope the authors will response the above questions in the rebuttal and include these in the final version. Given the novelty of the approach and extensive experiments, the submission would be stronger if the authors could fix these flaws.


Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
Neural networks have many successful applications, while much less theoretical understanding has been gained. Towards bridging this gap, we study the problem of learning a two-layer overparameterized ReLU neural network for multi-class classification via stochastic gradient descent (SGD) from random initialization. In the overparameterized setting, when the data comes from mixtures of well-separated distributions, we prove that SGD learns a network with a small generalization error, albeit the network has enough capacity to fit arbitrary labels. Furthermore, the analysis provides interesting insights into several aspects of learning neural networks and can be verified based on empirical studies on synthetic data and on the MNIST dataset.

This paper analyzes the generalization performance of two-layer ReLU networks in an overparametrized regime. Specifically, the networks are trained with SGD over separable multiclass  problems. In such setting, the authors prove that the learned network has small generalization error with high probability. This result generalizes the result in [7], which only considers linearly separable data.   This paper is well organized and easy to follow. The proof sketch and discussion on insights make it easy to understand the technique in use and the implications of this work.   There is no doubt that the authors address an important problem. Explaining the generalization performance of deep learning is a very core task of the modern machine learning. But the contribution of this work seems incremental. The theoretical result of this paper generalizes the result in [7] by adopting a more general separable dataset. It is true that such structured (multiclass) data is closer to the practical scenario. While the truly practical scenarios normally involve non-separable dataset, which cannot be easily analyzed with the approach used in the paper (the separability parameter \delta > 0 seems necessary even for the simplified case in Section 5).   Moreover, mixing the generalization and optimization in the main theorem likely complicates the analysis and restricts the scope of the result. In particular, SGD might have nothing special in regard to learning a solution with good generalization performance. It just happens to be the only practical method for neural network training. For random forest [a] and kernel machine [b], solutions that are not learned by SGD (e.g. exact solution for kernel regression) can achieve both (almost) zero training error and excellent generalization performance. Note that all these models/architectures are overparametrized. So we see empirically that for several overparametrized architectures, methods other than SGD can learn a solution with a small generalization error. Thus, it is hard to tell which inductive bias is more important to obtain the good generalization performance, the one derived from the optimization method or the one enforced by the architecture selection.    Some minor things:   Line 123: or every \epsilon ---> for every \epsilon.   Line 141: by SGD ---> by GD.   Line 276: converges to 0 ---> converges to 100%.   [a] Wyner, Abraham J., et al. "Explaining the success of adaboost and random forests as interpolating classifiers."  [b] Belkin, Mikhail, et al. "To understand deep learning we need to understand kernel learning."   ===== After Rebuttal ===== After reading the rebuttal and all reviews, I feel that this paper deserves a higher score.


Differentially Private k-Means with Constant Multiplicative Error
We design new differentially private algorithms for the Euclidean k-means problem, both in the centralized model and in the local model of differential privacy. In both models, our algorithms achieve significantly improved error guarantees than the previous state-of-the-art. In addition, in the local model, our algorithm significantly reduces the number of interaction rounds. Although the problem has been widely studied in the context of differential privacy, all of the existing constructions achieve only super constant approximation factors. We present-for the first time-efficient private algorithms for the problem with constant multiplicative error. Furthermore, we show how to modify our algorithms so they compute private coresets for k-means clustering in both models.
: Private algorithms for k-means. Here n is the number of input points, k is the number of centers, and d is the dimension. For simplicity, we assume that input points come from the unit ball, and omit the dependency in ε, as well as logarithmic factors in k, n, d, β, δ, from the additive error.
Before describing our new results, we define our setting more precisely. Consider an input database S = (x 1 , . . . , x n ) ∈ (R d ) n containing n points in R d , where every point x i ∈ S is the (sensitive) information of one individual. The goal is to identify a set of k centers C = {c 1 , . . . , c k } in R d approximately minimizing the following quantity, referred to as the cost of the centers
The privacy requirement is that the output of our algorithm (the set of centers) does not reveal information that is specific to any single individual. Formally, Definition 1.1 ( [12]). A randomized algorithm A : X n → Y is (ε, δ) differentially private if for every two databases S, S ∈ X n that differ in one point, and every set T ⊆ Y , we have Pr[A(S) ∈ T ] ≤ e ε · Pr[A(S ) ∈ T ] + δ.
Combining the utility and privacy requirements, we are seeking for a computationally efficient differentially private algorithm that identifies a set of k centers C such that w.h.p. cost S (C) ≤ γ · OPT S +η, where OPT S is the optimal cost. We want γ and η to be as small as possible, as a function of the number of input points n, the dimension d, the number of centers k, the failure probability β, and the privacy parameters ε, δ.
We remark that a direct consequence of the definition of differential privacy is that, unlike in the non-private literature, every private algorithm for this problem must have additive error η > 0. In fact, if all points reside with the d-dimensional ball, B(0, Λ), of radius Λ around the origin (as we assume in this paper) then η must be at least Λ 2 . To see this, consider k + 1 locations p 1 , . . . , p k+1 at pairwise distances Λ, and consider the following two neighboring datasets. The first dataset S 1 contains n − k + 1 copies of p 1 , and (one copy of) p 2 , . . . , p k . The second dataset S 2 is obtained from S 1 by replacing p k with p k+1 . Since in both cases there are only k distinct input points, the optimal cost for each of these datasets is zero. On the other hand, by the constraint of differential privacy, the set of centers we compute essentially cannot be affected by this change. Therefore we must have expected error of Ω(Λ 2 ) on at least one of these inputs. To simplify the presentation we assume that Λ = 1 in rest of the introduction.
Traditionally, in the non-private literature, the goal is to minimize the multiplicative error γ, with the current state-of-the-art (non-private) algorithm achieving multiplicative error of γ = 6.357 (with no additive error) [2]. In contrast, in spite of the long line of works on private k-means [8,28,14,18,27,33,31,32,17,6,29,21], all of the existing polynomial time private algorithms for the problem obtained only a super constant multiplicative error. We present the first polynomial time differentially private algorithm for the Euclidean k-means problem with constant multiplicative error, while essentially keeping the additive error the same as in previous state-of-the-art results. See Table 1 for a comparison.

The paper considers k-means under differential privacy. The main idea of differential privacy (applied to k-means) is that changing one point in the input should not significantly change the centers that are computed. This immediately means that any approximation algorithm for the differentially private k-means problem needs to have additive error: The introduction includes a simple example of points in k locations (=> optimum has zero cost). Now when moving one of these points to a different location, the centers may not change significantly; ergo one of the solutions has to have a large additive error; basically an error of D^2 where D is the largest pairwise distance. Thus, a common approach for this problem is to compute an approximation that is both multiplicative and additive. The best known result prior to this paper achieves a quality of O(k)*OPT + O~(d^0.51+k^1.51).  The paper dramatically reduces this guarantee to O(1)*OPT + O~(k^1.01 d^0.51 + k^1.5). The new algorithm works both in the centralized variant where a trusted server provides anonymization, and in the local variant, where the users themselves randomly change their point before sending it to an untrusted server. The results look very good to me; better multiplicative approximation, no increase in the additive term, and in the local model, the number of rounds is also decreased. My only concern is whether the paper really fits NIPS, because a) not very much of the proof makes it into the actual paper, and b) there are no experiments. It's not clear to me how practical the algorithm is; the main routine does not look to complicated, but the subroutine might be. Aside from the missing experiments, this is clearly the best paper in my (not necessarily representative) batch. I do not know whether a paper without experiments can appear at NIPS; I'll vote for acceptance and wait for the other reviewer's opinion on that.  The introduction is very well written. The problem definition becomes clear immediately, and the need for additive approximation is explained nicely. The best non-private approximation for k-means is cited. I'm no expert on differentially private k-means algorithms, but the related work section wrt that topic looks solid to me as well. I appreciated the paragraph about classical algorithm not being private, and the pointer to the k-means++ adaptation. The technical contribution is also made very clear; previous work is summarized clearly. The paper builds upon a previously obtained result which essentially gives a differentially private algorithm for k-means if and only if one can supplement it with a set of good center candidates of poly(n) size. At first glance, this seems easy, since for non-private k-means, one can simply take the input points as the candidate set, this is of size n and has an error of 2 compared to centers from R^d. However, for private k-means clustering, this is not possible -- the centers can not be a subset of the input points, because otherwise, removing a point may significantly change the computed centers. The paper has two main contributions. First, it shows how to compute a good center set; this part is based on a different related paper, but seems refined. Second, it embeds both the candidate set computation and the subroutine that computes good centers given such a candidate set into a larger main algorithm. Roughly, this algorithm computes a candidate set, uses it to compute centers, and then eliminates points which it now deems "covered" by the new centers. This process is then repeated log log n times, and then the probability that all points are covered is sufficiently high.  A question for the author feedback: Just for my clarification: How does the fact that we can not pick S as the candidate center set show in Theorem 2.1? What if I just hand the algorithm in [16,6] Y=S, how does reading the theorem tell me that that does not lead to a differentially private algorithm?  Smaller remarks: - In Table 1, the ~ notation hides log n - factors. As far as I thought, the O~(X) is usually used to hide factors that are polylogarithmic in X -- but n is not a part of the expression here. It is stated in the caption of Table 1 that logarithmic factors in n are omitted; yet I think the dependence on log n should be stated somewhere in the introduction in the text (if I missed it, sorry). - The last sentence on page 3 (line 112+113) confused me quite a bit. It might be a good point in the paper to describe that one can not simply take the input points as a candidate set. This is obvious once observed, but when usually working on non-private k-means, the statement that an O(1)-approximation would work for a discrete set and is simply not applicable in R^d just sounds strange. - The citations for non-private coresets in line 178 seem a bit arbitrary; they should probably contain at least Dan Feldman, Michael Langberg: A unified framework for approximating and clustering data. STOC 2011: 569-578 which is to my knowledge the currently best for k-means. - Theorem 2.1: "a finite set centers" -> "a finite set of centers" - The sentence above Lemma 3.5 that the proof is omitted seems to be not true, since the proof is there.  Edit after author feedback: I like this submission, and this has not changed.


MetaReg: Towards Domain Generalization using Meta-Regularization
Training models that generalize to new domains at test time is a problem of fundamental importance in machine learning. In this work, we encode this notion of domain generalization using a novel regularization function. We pose the problem of finding such a regularization function in a Learning to Learn (or) metalearning framework. The objective of domain generalization is explicitly modeled by learning a regularizer that makes the model trained on one domain to perform well on another domain. Experimental validations on computer vision and natural language datasets indicate that our method can learn regularizers that achieve good cross-domain generalization.

** Post Rebuttal ** The rebuttal adequately addresses all of my suggestions and questions. All of the new experiments in the rebuttal are valuable and should be included in the final. I recommend accepting the paper, assuming that the authors will add their new results to the paper.  ** Original Review ** This paper proposes a meta-learning approach to domain generalization, where the method learns a regularization term that is added to the task learning objective. Once the regularizer has been learned across different domains, the paper proposes to train on all domains using this regularizer. The paper additionally proposes to have a feature network that is shared across domains/tasks to enable more scalable meta-learning. The method provides a small, but significant improvement over simply training a model on all tasks, as well as prior domain generalization methods. The experiments provide additional ablations and visualizations for better understanding the method.  Pros: The paper addresses an important problem of the ability to generalize well to new domains given no data in the new domain, and provides an innovative approach based on learning to regularize to the learning process. The idea of learning a regularizer is interesting and relevant beyond the particular domain generalization problem setting addressed in this paper, increasing the potential impact of the paper. The experiments and analysis are thorough.  Cons: The empirical benefit provided compared to baseline models is relatively small. The clarity of the paper could be improved, both in terms of notation and how it is presented. Further, the Bayesian interpretation should be further fleshed out for it to be meaningful. There are details of the algorithm that are not included in the current version that would make it difficult to reproduce. [Detailed feedback below]  *More detailed comments & feedback* Related work: The idea of sharing a feature network across domains has been recently proposed in the meta-learning literature [2]. It would be good to reference [2] in the paper, for the sake of keeping track of relevant work. But, I do not penalize the authors because [2] is not published and came out ~3 months before the deadline. The related work has a detailed discussion of how this method relates to MLDG, but does not discuss the relation to other works. It would be helpful to include such discussion rather than simply summarizing prior methods.  Bayesian interpretation: The Bayesian interpretation doesn't completely make sense. First, what is the graphical model underlying this model? It would be helpful to have a Figure that illustrates the graphical model. Is it the same as the graphical model in [1]? [If so, it might be nice to reconcile notation with that prior work, with theta as the prior parameters and phi as the task parameters.] Second, why does a point estimate for phi make sense? How does the computation of this point estimate relate to the Bayesian interpretation? Prior work [1] developed a Bayesian interpretation of MAML using a MAP estimate to approximate an integral, drawing upon a previous connection between gradient descent and MAP inference. A MAP estimate provides a crude, but somewhat sensible approximation, given that the point has high probability. But, in this case, no such reasoning seems to exist. Discussing the connection to [1] would also be useful.  Comments on Clarity: 1. An algorithm for what happens at meta-test time would be really helpful. How are the task parameters theta_g initialized? (randomly or using the existing theta_i's?). How many gradient steps are taken at test time? Is it the same as l? Is both psi and theta updated, or just theta? 2. It would be helpful to have a diagram that illustrates the entire meta-learning approach. Figure 1b is a step in the right direction, but it would be nice to visualize the update to phi and to include multiple domains/thetas. The caption of Figure 1 is thorough, which is very nice. But, it doesn't describe how phi is used.  Algorithmic+Experimental suggestions and questions: 1. Right now, pairs of tasks are selected for training the regularizer. Since, at meta-test time, the model will be trained with the regularizer across multiple domains, it seems like it might make more sense to have meta-training of phi mimic this setting more closely, having each theta_i being trained across a set of domains, e.g. all but one domain, rather than an individual domain, and then training phi on domains not in that set. Right now, phi is trained only for thetas that were trained on an individual domain rather than multiple domains. 2. Would it make sense to have a replay memory of previous thetas? It seems like phi might learn to be a good regularizer for parameters that have been trained on a task for a long time (and might forget to be a good at regularizing randomly initialized parameters). 3. What are the implications of freezing the feature network? An ablation with varying amounts of layers in the feature network vs. task network could help study this question empirically. 4. In the experiments, it would also be useful to plot phi_i. 5. Are there experimental settings where we would expect more improvement? Perhaps in settings with more domains?  More minor comments: - a more common notation for a set of parameter vectors is to use uppercase \Theta (vs. \tilde{\theta}). - A lot of notation is definied in section 3.1 and some of it is not used later, such as M, T, \tilde{\theta}, \mathcal{D}. It might be better to consolidate some of this notation. - "generalization to a new domain is not guaranteed. To accomplish this, we propose...": This wording suggests that the proposed approach guarantees generalization to new domains, but theoretical guarantees are not discussed. - The last paragraph of section 3.1 is confusing because it isn't clear if there is one theta or multiple theta_i. When referring to theta, does the paper mean {theta_1, ..., theta_p} or an individual theta_i or theta_g? - In equation 4 and elsewhere, it would be helpful to put brackets around everything that is inside the gradient operator, to make it clear that the regularizer term is inside the gradient operator.  [1] Grant et al. ICLR 2018 [2] Zhou et al. http://arxiv.org/abs/1802.03596


Bayesian Control of Large MDPs with Unknown Dynamics in Data-Poor Environments
We propose a Bayesian decision making framework for control of Markov Decision Processes (MDPs) with unknown dynamics and large, possibly continuous, state, action, and parameter spaces in data-poor environments. Most of the existing adaptive controllers for MDPs with unknown dynamics are based on the reinforcement learning framework and rely on large data sets acquired by sustained direct interaction with the system or via a simulator. This is not feasible in many applications, due to ethical, economic, and physical constraints. The proposed framework addresses the data poverty issue by decomposing the problem into an offline planning stage that does not rely on sustained direct interaction with the system or simulator and an online execution stage. In the offline process, parallel Gaussian process temporal difference (GPTD) learning techniques are employed for near-optimal Bayesian approximation of the expected discounted reward over a sample drawn from the prior distribution of unknown parameters. In the online stage, the action with the maximum expected return with respect to the posterior distribution of the parameters is selected. This is achieved by an approximation of the posterior distribution using a Markov Chain Monte Carlo (MCMC) algorithm, followed by constructing multiple Gaussian processes over the parameter space for efficient prediction of the means of the expected return at the MCMC sample. The effectiveness of the proposed framework is demonstrated using a simple dynamical system model with continuous state and action spaces, as well as a more complex model for a metastatic melanoma gene regulatory network observed through noisy synthetic gene expression data.

POST-REBUTTAL:  Thank you for the clarifications. I've increased the overall score because the rebuttal made me think of this work as an interesting tradeoff between generality and the time when when computational cost investment has to be made (online vs. offline). At the same time, BAMCP has the significant advantage of handling more stochastic (as opposed to nearly deterministic, as in this paper) scenarios, and I would encourage you to work on extending your approach to them.   ==============================================================  The paper proposes a Bayesian RL method for MDPs with continuous states, action, and parameter spaces. Its key component is efficiently approximating the Q-value function expectations over the posterior belief over the parameter space, which is achieved by a combination of  GPTD during the offline stage of the algorithm and MCMC during the online stage. The approach is evaluated against MMRL and greedy myopic action selection on two problems, one of which is a synthetic MDP and another is a problem of computing gene expression interventions for a regulatory network.  To me, the paper's main novelty is the technique for approximating Q-value function expectations over parameter posterior given approximate Q-values for a sample of parameter values drawn from the prior. The approach has quite a few moving parts, and it's impressive to see them work in concert.  At the same time, the paper would benefit greatly from a better explanation of its motivation/applicability and, on a related note, from crisper positioning w.r.t. related work. The lack of clarity in these aspects makes it difficult to determine potential existing alternatives (if any) to the proposed algorithm and its parts. More specifically:  - Motivation. The submission's promise of handling decision-making problems with uncertain dynamics and continuous parameter spaces in data-poor environments isn't unique to the approach presented here. It is the motivation for Bayesian RL in general, and different BRL methods deliver on it varying degrees of success. The question is: what exactly distinguishes the proposed approach's capabilities from prior art in BRL? Is it the ability to handle continuous state spaces? This is a bit hard to believe, because given a sample of model parameters, i.e., a fully specified MDP, there is a range of methods that can solve such MDPs; discretizing the state space is one option. Same goes for continous action spaces. In fact, I don't see why this paper's method would have particular advantages in this regard, because despite its ability to approximate Q-value expectations, it doesn't offer anything special to facilitate action selection based on these Q-value estimates -- a non-trivial problem in itself when you use critic methods in continuous state spaces. In short, from the paper's current description, I couldn't deduce the advantages of the proposed method (or the problem characteristics where its advantages are especially noteworthy).  - Positioning w.r.t. related work. Currently, the paper seems to mention only one three BRL works, and they are really dated by now. Exactly what other algorithms the proposed work should be compared to depends on its motivation, but one algorithm that is almost certainly relevant is BAMCP:  Guez, Silver, Dayan. "Scalable and Efficient Bayes-Adaptive Reinforcement Learning Based on Monte-Carlo Tree Search". JAIR, 2013.  Its vanilla version described above assumes discrete state and action spaces, but using discretization this restriction can be lifted, and its derivatives may have done so already. In general, there has been a lot of BRL work since references [7], [8], and [9], as described in the following survey:  Ghavamzadeh, Mannor, Pineau, Tamar. "Bayesian Reinforcement Learning: A Survey". Foundations and Trends in Machine Learning, 2015. Newer versions are on arXiV.  It would be good to see the related work coverage expanded to at least some of the recent BRL methods described there.    More technical comments:  - The phrase "uncertain dynamics" is somewhat unfortunate, because it can be easily misinterpreted to refer to dynamics that are probabilistic, while this paper seems to handle only the cases where the dynamics itself is near-deterministic but unknown.  - The claim on lines 108-113 makes Q-value computation for a known \theta looks harder than it is. Every known \theta essentially encodes an MDP with known dynamics (and known reward function -- this appears to be assumed in the paper). Calculating the Q-function for this MDP is a planning problem, which can be solved in a wide variety of ways, which can be very efficient depending on the MDP structure. In particular, I'm not sure why deep learning is necessary here. Is this because you assume continuous states and actions and a DNN-based Q-value approximator?  - Is Equation 3 missing the expectation sign? If not, it is missing something else, because, as stated, the value of the left-hand side depends on a particular trajectory sampled from \pi, and the LHS value is ill-defined without conditioning on this trajectory.  - I'm confused by Section 3.1. Its stated goal is computing (an approximation of) Q^*_{\theta}, the Q-value function of the optimal policy under MDP parameters \theta. However, then it goes on to explain how to compute Q^{\pi}_{\theta} for policy \pi from Equation 10. What's the relationship between \pi from Equation 10 and the optimal policy under \theta? Line 147 talks about the exploration-exploitation tradeoff -- why do you need to resolve it if you are doing all of this for a _specific_ MDP parameter vector \theta?  - The paper switches between subscript and superscript indexes for \theta. It appears to be using subscript indexes for \thetas sampled from the prior (as on line 175) and superscript indexes for \thetas sampled from the posterior (as in Equation 14). Is this the distinction the paper is trying to make? If so, I suggest keeping it uniform and using, e.g. \theta^{prior}_i and \theta^{posterior}_i instead.  - Are there any hypotheses as to why the proposed algorithm outperforms MMRL on the synthetic MDP?  - To beef up the the experimental evaluation, I'd replace the synthetic MDP with something more complicated.   Language errors:  "the sequence of observed joint state and action simulated" --> "the sequence of observed joint state and action pairs simulated"  "specially in regions" --> "especially in regions"   IN THE REBUTTAL, please comment on the motivation/advantages of the proposed algorithms w.r.t. others, particularly BAMCP and its derivatives, and cover the questions in the "technical comments" section of the review, as space allows.


Mesh-TensorFlow: Deep Learning for Supercomputers
Batch-splitting (data-parallelism) is the dominant distributed Deep Neural Network (DNN) training strategy, due to its universal applicability and its amenability to Single-Program-Multiple-Data (SPMD) programming. However, batch-splitting suffers from problems including the inability to train very large models (due to memory constraints), high latency, and inefficiency at small batch sizes. All of these can be solved by more general distribution strategies (model-parallelism). Unfortunately, efficient model-parallel algorithms tend to be complicated to discover, describe, and to implement, particularly on large clusters. We introduce Mesh-TensorFlow, a language for specifying a general class of distributed tensor computations. Where data-parallelism can be viewed as splitting tensors and operations along the "batch" dimension, in Mesh-TensorFlow, the user can specify any tensor-dimensions to be split across any dimensions of a multi-dimensional mesh of processors. A Mesh-TensorFlow graph compiles into a SPMD program consisting of parallel operations coupled with collective communication primitives such as Allreduce. We use Mesh-TensorFlow to implement an efficient data-parallel, model-parallel version of the Transformer [16] sequence-to-sequence model. Using TPU meshes of up to 512 cores, we train Transformer models with up to 5 billion parameters, surpassing state of the art results on WMT'14 Englishto-French translation task and the one-billion-word language modeling benchmark. Mesh-Tensorflow is available at https://github.com/tensorflow/mesh .
collective communication primitives such as Allreduce. We use Mesh-TensorFlow to implement an efficient data-parallel, model-parallel version of the Transformer [16] sequence-to-sequence model. Using TPU meshes of up to 512 cores, we train Transformer models with up to 5 billion parameters, surpassing state-of-the-art results on WMT'14 English-to-French translation task and the one-billion-word Language modeling benchmark.

Given the author's additional information about their plans to open-source the language described in this paper, I've upgrade my review. I'm still concerned that the original paper didn't mention anything about this intent to open-source; but I'm willing to give them the benefit of the doubt because if this is open-sourced, it could be very helpful to the community.  Here are some minor typos I found: Capitalize “equation”, “table”, “section” as needed throughout the paper. Line 112: “pleasant” to “pleasantly” (and throughout the paper where needed) 219: “tpu” to “TPU” 222: “(3)” to “Table 3”  ORIGINAL REVIEW This paper introduces a convenient notation for compactly describing mappings from a tensor representation of a learning problem’s data and model parameters to an n-dimensional mesh of compute nodes.  The notation is flexible enough to capture both data-parallel and model-parallel mappings, as well as mixtures of the two. The paper uses these mappings to derive first order approximations to the computation and communication times required to perform a single backpropagation update (assuming an optimized communication library like MPI).  The paper then uses these estimates to develop a strategy for simultaneously growing the model size and the number of compute nodes used such that communication does not become a major performance bottleneck.  The paper then uses this strategy to explore the parallel performance of the Transformer sequence transduction model.  The strategy allowed the authors to train a 5B parameter system on 256 nodes and get state-of-the-art performance on the WMT14 En-Fr data set.  When I first read this paper, I thought that the authors meant that they had created a new programming language for easily and efficiently mapping machine learning problems to parallel computer nodes.  I was hoping that this “language” would be made available to the reader and that it would include some automatic optimization of the mappings; but as I read more of the paper, I realized that the paper only presented a notation for representing the mapping.  If a new programming  language (or tool) has been created, the authors should make that more clear and maybe give pointers to it.  If it is only a notational convenience, the authors should make that more clear too.  If the contribution of Section 4 is only to introduce a notational convenience, the authors should explain the significance more clearly and/or concretely.  For example, the notation in Algorithm 1 is simply Einstein Summation Notation in another form.  It doesn’t add significantly to the reader’s understanding of the problem.  Furthermore, the observations made in Table 1 are fairly well-known from a parallel computing point of view - most practitioners realize this computation/communication trade off.  And for those NIPS readers who don’t, the paper has no explanation for where these numbers come from.  The paper could be improved if Section 4 was much shorter and the authors spent time explaining how they calculated the terms in Table 1. It seems that the major result of this paper is a better BLEU score from the ability to run a bigger model.  I’m not sure that is significant enough.  The paper would benefit from clarification in several places.  For example, “b”, “d_x” and “d_y” are never explicitly defined.  Equation (3a) is described and then never used again in the paper.  The paper’s “{b-->0}” notation is never explicitly defined.  Presumable “0” means dimension zero of the compute mesh; but it would be better it the authors specifically said it so that the reader doesn’t have to guess.  Likewise, “F” and “D(X,M)” are never defined.  (I know “F” is an arbitrary function, but the paper should say that instead of making the reader guess.) Also, for example, when two model dimensions map to a single mesh dimension, does that mean the mesh dimension is split 50/50?  It could be split based on some communication minimization, or other, algorithm.  The reader has to guess.  The text mentions four layouts but only shows three in Table 1.  In Table 3, why are two of the BLEU scores missing?  Was it that they couldn’t run?  Ran too slow? Wasn’t enough time? That should be explain.  In Section 5, the reader would benefit from some pictures showing, for example, the matrix multiplication, etc., to help them “see” the mapping.  “SOTA” should be replaced with “state-of-the-art” throughout the paper.


Learning semantic similarity in a continuous space
We address the problem of learning semantic representation of questions to measure similarity between pairs as a continuous distance metric. Our work naturally extends Word Mover's Distance (WMD) [1] by representing text documents as normal distributions instead of bags of embedded words. Our learned metric measures the dissimilarity between two questions as the minimum amount of distance the intent (hidden representation) of one question needs to "travel" to match the intent of another question. We first learn to repeat, reformulate questions to infer intents as normal distributions with a deep generative model [2] (variational auto encoder). Semantic similarity between pairs is then learned discriminatively as an optimal transport distance metric (Wasserstein 2) with our novel variational siamese framework. Among known models that can read sentences individually, our proposed framework achieves competitive results on Quora duplicate questions dataset. Our work sheds light on how deep generative models can approximate distributions (semantic representations) to effectively measure semantic similarity with meaningful distance metrics from Information Theory.

The paper addresses the problem of learning the representation of questions/sentences as an input unit. Like word2gauss, the embedding is into distributions (as opposed to a single point of a vector space) and in particular into Gaussian distributions. Embedding into a distribution enables the use of a transport distance, such as W2 for computing discrepancy between distributions which has advantages over alternatives such as KL divergence but is expensive to compute. Here the specific constraints on the distributions made the computation of the transport distance efficient. The application considered is question de-duplication. The authors solve the problem in two separate stages. The first stage is the actual representation learning stage, which embeds questions into the Gaussian distribution. The second stage is to learn from labeled data to decide if the two pre-trained representations correspond to equivalent questions or not. For the first stage a generative (LSTM based) VAE model is proposed to convert a question into itself or a similar question. For the second stage a Siamese structure is proposed that is based on W2 distances. The model is evaluated on Quora question deduplication dataset and performed strongly.   I liked this paper: there is a creative component in the overall proposed solution and the architecture and the pieces fit together well. The embedding of a whole sentence to a distribution seems very natural, W2 seems like a good way to compute distances, etc. I also liked the idea of "similar-encoder", which could also be useful as opposed to "auto-encoder", as an augmentation method. The two stage development of the solution raises my curiosity though. The first stage, i.e. text generation, is a difficult task. Eliminating this stage completely and performing an end-to-end training would have been an alternative way to go. The pre-training from the first stage could still be used for initialization, but the question representations could have been fine-tuned for the actual task. I was hoping to see more discussions/results on that alternative. Moreover, it is surprising that no generation result is reported.     The paper is well organized and readable. But it needs proofreading and improvement in presentation in places. Mathematical notations should be more carefully used. - The notations for vectors and scalars seem to be mixed up.  See for example Section 3.2. There seem to be a dimension mismatch in Equations (3) and (4). The notation for matrix square root is unclear.  - Section 4.2 is not clear. What is "element-wise" Wasserstein? Is it not that Wasserstein distance returns a number (scalar) for a pair of vectors? Is the dimension of the input to the next module h+1? (One for the W norm and h for the Hadamard product?) - Not enough details is given about the W2 computation for Gaussians. In particular, the first paragraph of page 6 (in Section 4.1) lacks enough details and is cryptic. Needs expansion. - Minor:  "Similarly to","Sigm", "sentences representation", "euclidean", "gaussian", "figure 3", "[3][4][5][6] and [7]", "87,5".  The approach proposed is novel to my knowledge. The work is a good continuation of the line of research that focuses on embedding into distributions. It is also a good application of W2 distance between distributions. The generation part of the proposal ( "Repeat-Reformulate") reminds me of "Sequence to better sequence" in [1]. Citation might be a good idea.  The significance of the method is enough to be accepted I think. Learning representation for a question/sentence is an important and unsolved problem. Embedding to the right mathematical object is an important question and explorations and insights on that seems useful. Coming up with a robust distance between distributions that is easy to compute is also beneficial. Finally, the application considered has received attention lately in ML and NLP community.  Overall, considering strengths and weaknesses of the paper, I vote for the paper to be accepted.  ---------------------------- [1] Mueller, J., Gifford, D., Jaakkola, T.  "Sequence to better sequence: continuous revision of combinatorial structures",  in ICML 2017. ---------------------------- Thanks for the clarification on the "element-wise" Wasserstein operation that is mentioned. It makes more sense now. That should definitely be clarified in the write up. (Mathematically defining the ith element of the operation by using indices would be helpful to clarify the definition.)


Geometry Based Data Generation
We propose a new type of generative model for high-dimensional data that learns a manifold geometry of the data, rather than density, and can generate points evenly along this manifold. This is in contrast to existing generative models that represent data density, and are strongly affected by noise and other artifacts of data collection. We demonstrate how this approach corrects sampling biases and artifacts, thus improves several downstream data analysis tasks, such as clustering and classification. Finally, we demonstrate that this approach is especially useful in biology where, despite the advent of single-cell technologies, rare subpopulations and gene-interaction relationships are affected by biased sampling. We show that SUGAR can generate hypothetical populations, and it is able to reveal intrinsic patterns and mutual-information relationships between genes on a single-cell RNA sequencing dataset of hematopoiesis.

It is the difficult to discover small clusters from imbalanced dataset in classification/clustering tasks. The authors try to tackle this problem by imputing new data points into the original dataset using a diffusion map based approach. Diffusion map is a Gaussian kernel normalized by a diagonal matrix D, whose diagonal elements are the total connectivity of the corresponding data point. This kernel can be further extended to measure-based Gaussian Correlation (MGC) kernel, which only considers a set of reference points. The main idea is to generate new data points using the MGC kernel in neighbourhood of each original data point and combine them for downstream classification/clustering tasks.  The authors then discusses how to select the bandwidth of the Gaussian kernel, how many new data points to generate for each original data point, how to set the number of multiplication steps of the Gaussian kernel. After that the authors provide a proof about setting l(i) which I do not go through.  The MNIST experiment shows the method is able to interpolate data points in low density regions, while variational autoencoder is not. After that the authors check the generated data points are uniformly distributed in the manifold. Then they show SUGAR achieves better performance compared with other data sampling methods on classification and clustering of imbalanced datasets. In the final single cell dataset, they show SUGAR helps to separate different cell types from the data.  In general I feel the key strength of SUGAR comes from the random walk near the original data points, which may help to connect the low density regions. The exploration property comes from the diffusion distance (line 141). The larger t used in the diffusion distance, the closer the newly generated data points to the original. While solving the minor cluster problem, I have a gut feeling imputing new data points may bring up new problems such as fake new clusters.  I have some questions regarding the experiments: 1. MINST: this experiments may be too simple. Only one "6" is rotated to generate the whole dataset, so there is no noise in principle. Comparison with GAN instead of VAE may be more realistic. In Figure 1, how is t determined? 2. Clustering of imbalanced data: Figure 3a shows the contribution of SUGAR is merging the two clusters in "S" but not imputing the low density regions in "G" (bottom part), what is the t used here? How is K chosen in k-means for the newly generated dataset? 3. Biological data: Figure 4d, what are the x-axis and y-axis? 4. what is "f" in line 133?  Although with flaws, I feel this is a nice paper to be accepted.   Updates: Based on the authors' feedback, it seems to be a nice paper.