Learning Continuous Control Policies by Stochastic Value Gradients
We present a unified framework for learning continuous control policies using backpropagation. It supports stochastic control by treating stochasticity in the Bellman equation as a deterministic function of exogenous noise. The product is a spectrum of general policy gradient algorithms that range from model-free methods with value functions to model-based methods without value functions. We use learned models but only require observations from the environment instead of observations from model-predicted trajectories, minimizing the impact of compounded model errors. We apply these algorithms first to a toy stochastic control problem and then to several physics-based control problems in simulation. One of these variants, SVG(1), shows the effectiveness of learning models, value functions, and policies simultaneously in continuous domains.

The paper introduces a formalizm which enables that allows extending value-based      policy gradient methods to stochastic policies in stochastic     environments. The formalism is based on reparametrization (modeling     stochasticity as a deterministic function of exogenuous noise).     The paper presents algorithms based on this formalism and test them on      multiple control problems.      The problem of extending value-gradient methods to stochastic policies     and environments seems important. The experimental part of the paper      covers different challenging control domains. The main issue with the      paper is that the presentation needs significant improvement to be accessible     to a general RL audience, which is a reasonable requirement from an RL     paper. Specifically, terms are used without being defined first, many     descriptions are informal where formality is required, and many critical     details are missing.This makes it hard to follow and evaluate the exact     contributions of the paper for people outside the community that works on      RL for robotics in continuous state/action spaces.      Details:     - the abstract is currently a bit misleading: it claims to apply     algorithms to "a range of difficult physical control problems", without     mentioning that they are *simulated* control problems. The authors should     clarify this point explicitely in the abstract in a final version of the     paper, if accepted.       line 16 - long sentence, needs break.      line 38 - what do you mean by 'chain' - please be more formal      line 40 - 'backpropagating' has different meanings - please     explain which one of them you refer to here. What is being backpropagated?      line 41 - 'replace the derivatives' - replace them where?      line 51, 55 - what do you mean by 'unroll', please clarify      line 56 - 'learned jointly' - why is it new, many model-based     algorithms can be considered to learn jointly the model,     value-function and policy      line 57 - long sentence, break      line 60-61 - the sentence is vague - please be more explicit      line 76 - Define MDP as a tuple (S, A, P, R, gamma, s0)      line 78 - please clarify why learning time-invariant policies if     reward is time-varying      line 82 - define gamma mathematically (instead of 'strictly less than...')      Equation 1 - missing upper limit in the summation      line 92-94 - definition of V' is confusing, also you might want to use ':="      line 105 - you differentiate with respect to state and theta (not action)      Eq. 3-4 - please provide more details on how you perform the derivation here      line 108 - 'purely model-based formalism' expression is used but was never defined here      line 108 - earlier, did you mean by 'backpropagating' to     backpropagation of the derivatives described here? if yes,     please say explicitely      line 111 - 'unrolling' was never defined      lines 111-114 are unclear      line 117 - 'model' used but never defined      line 118 - how do you define 'modeling errors' and how     sensitive V_theta to them ('highly sensitive' is not informative     enough)      line 119 - 'we start instead' - instead of what?      line 122 - 'real, not planned' - what do you mean by 'planned' - please define      line 124 - 'critics' used but never defined in this context      lines 125-127 - sentence is unclear      line 131 - 'backpropagate efficiently' - what do you mean by saying     that: what is your efficiency measure      line 136 - 'we consider conditional densities' - what do you mean by 'consider'      line 148 - what do you mean by 'exploits' the Jacobian      line 165 - 'standard planning method' - can you provide reference?     please define 'planning' in this context?      Eq 10-11 - you use notation that is not explained: what is '|_\eta,\ksi'      line 185 - 'we can now replace..' - why you can do that?      line 186 - why this case is 'important'      line 187 - \Sigma was not defined      Sec 4.2 - Since it doesn't contain any details, consider moving it to     forward near the place you provide more details      line 218 - 'the parameters of the critic' used but were never defined here      line 257 - 'our framework easily supports' - what do you mean by 'easily'      Sec 6 - it would be good to mention a relevant figure first (e.g. Fig 2) and      then describe in detail what it shows.      Section 6.1 - based on what you determine that [25] is     state-of-the-art? also is it state-of-the-art on a given domain?     multiple domains?


The Return of the Gating Network: Combining Generative Models and Discriminative Training in Natural Image Priors
In recent years, approaches based on machine learning have achieved state-of-theart performance on image restoration problems. Successful approaches include both generative models of natural images as well as discriminative training of deep neural networks. Discriminative training of feed forward architectures allows explicit control over the computational cost of performing restoration and therefore often leads to better performance at the same cost at run time. In contrast, generative models have the advantage that they can be trained once and then adapted to any image restoration task by a simple use of Bayes' rule. In this paper we show how to combine the strengths of both approaches by training a discriminative, feed-forward architecture to predict the state of latent variables in a generative model of natural images. We apply this idea to the very successful Gaussian Mixture Model (GMM) of natural images. We show that it is possible to achieve comparable performance as the original GMM but with two orders of magnitude improvement in run time while maintaining the advantage of generative models.

The paper proposes a fast approximate inference approach for MAP estimation with the generative EPLL method [21]. To that end, discriminative training is used to replace the computationally expensive update step of the latent variables during alternating optimization. The experimental results show that while being 1-2 orders of magnitude faster, this approximate inference procedure causes only a relatively modest drop in performance as compared to previously-used ("exact") EPLL inference. # Positive I very much agree with the agenda of the paper to retain the benefits of generative approaches, as discussed at the beginning of the paper. As far as I see, the paper unfortunately doesn't really propose a generic method, since it only addresses a very specific problem of inference with the EPLL method. However, it is an interesting approach to use discriminative methods to approximate expensive steps during generative inference. Have you thought about how this can be used more generally? This would really interest me.It is interesting that EPLL yields similar performance with the proposed approximative inference procedure compared to the "exact" one. This has value, since EPLL yields state-of-the-art results and can now also be applied to large images with the proposed inference method (Figure 7 is quite impressive); I think the paper should emphasize this more.The puzzling part to me is why EPLL is seemingly the only generative approach that is competitive with discriminative methods; this would be interesting to understand. # Negative I think that the paper over-promises its contributions. I was really excited in the beginning while reading the introduction, but was then somewhat disappointed at the end. This is because the paper is somewhat misleading due to introduction being very generic and promising a general method to combine generative and discriminative approaches. However, the paper doesn't propose a "deep" integration between generative and discriminative methods. It requires standard generative training of the image prior, then trains a discriminative predictor to approximate expensive steps of generative inference.Furthermore, the paper then addresses only a very specific problem for performing MAP estimation with the EPLL method, which is as far as I see not easily applicable beyond this. Moreover, other generative approaches with filter-based image priors don't have the problem that updating the latent variables is computationally expensive [e.g., Krishnan and Fergus, "Fast image deconvolution using hyper-Laplacian priors", NIPS 2009].Although not really important for the actual contribution of the paper, the discussion of discriminative and generative approaches starting in l. 111 is not very nuanced. There are other things to consider than efficiency of test-time inference and "modularity", such as generative approaches being able to use different loss functions at test time, or performing marginalization as a principled approach to handle unknown random variables. # Clarity - The explanation of the gating network in sec. 3 is hard to understand, I think it needs some work to make it more clear. - Table 1 doesn't say for which test data these denoising results have been obtained. Are these different test sets in each paper? - Typo in Eq. 7: the weight w_i^j should be w_i^k in the denominator. - Typo in l. 102: "Hel-Or used..." -> "Hel-Or and Shaked used..." - Figure 2 is not clear since the different parts of the figure are not labeled. Also note that there seems to be a missing connection in the gating network on the right side: the bottom left node should be connected to the middle right node. - I find the title "return of gating network" not really appropriate. Also, "combining generative models and discriminative training in natural image priors" indicates to me that a generic method is proposed in the paper, which is not the case. # Miscellaneous - Note that "modularity" can also be achieved with discriminative approaches by training with a likelihood component that can later be replaced, which has been done by [Chen et al., "Revisiting loss-specific training of filter-based MRFs for image restoration", GCPR 2013, doi:10.1007/978-3-642-40602-7_30]. They trained a CRF via loss-based training for image denoising and then used it without re-training for deconvolution and super-resolution by swapping the likelihood term. - Note that EPLL inference doesn't actually use half-quadratic splitting as introduced by Geman and colleagues (for filter-based MRFs), cf. [Geman and Reynolds, "Constrained restoration and the recovery of discontinuities", PAMI 1992; Geman and Yang, "Nonlinear image recovery with half-quadratic regularization", TIP 1995]. I think the used method is best characterized as a quadratic penalty method for approximate constrained optimization [cf. Nocedal and Wright, "Numerical Optimization", Sec. 17.1].


Rapidly Mixing Gibbs Sampling for a Class of Factor Graphs Using Hierarchy Width
Gibbs sampling on factor graphs is a widely used inference technique, which often produces good empirical results. Theoretical guarantees for its performance are weak: even for tree structured graphs, the mixing time of Gibbs may be exponential in the number of variables. To help understand the behavior of Gibbs sampling, we introduce a new (hyper)graph property, called hierarchy width. We show that under suitable conditions on the weights, bounded hierarchy width ensures polynomial mixing time. Our study of hierarchy width is in part motivated by a class of factor graph templates, hierarchical templates, which have bounded hierarchy width-regardless of the data used to instantiate them. We demonstrate a rich application from natural language processing in which Gibbs sampling provably mixes rapidly and achieves accuracy that exceeds human volunteers.

This paper develops a new property of hypergraphs that helps bound the mixing time in Gibbs sampling. Analyzing the mixing time is always a hard problem in MCMC, so this paper addresses an important issue in Gibbs sampling. The paper defines the notion of hierarchy width for factor graphs. The main result that the authors show is that, the mixing time is exponential in the hierarchy-width of the factor graph and the maximum difference in weights encoded by the potentials of the factor graph. The authors then show that for specific factor graph templates, the aforementioned results can be applied and the mixing time of Gibbs sampling can be bounded polynomially in the number of variables that instantiate the templates.The main novelty is theoretical results that show that for specific classes of template factor graphs, Gibbs sampling mixes rapidly.Figure 1 seems to come out of the blue. It is hard to understand this figure since almost everything in this figure is explained much later. One of the main problems with Gibbs sampling is mixing when there are 0 values (or near-determinism) in the potentials. I was not sure how your results would change in such cases. Are there some assumptions about the underlying distribution structure.What is the treewidth typically when factor graph templates are hierarchical? That is, can we have high treewidth models using these templates? If not, then it might not require approximate inference such as Gibbs sampling.I was a little confused by what the three semantics represent. That is, is there a drawback from using the logical/ratio semantics (where your results hold) over the linear semantics (where your results do not hold)On the real world applications, if only subgraphs are hierarchical do your results of rapid mixing hold? I assumed that you need the entire factor graph template to be hierarchical as in section 3.1. Does this happen often in practice?


Structured Transforms for Small-Footprint Deep Learning
We consider the task of building compact deep learning pipelines suitable for deployment on storage and power constrained mobile devices. We propose a unified framework to learn a broad family of structured parameter matrices that are characterized by the notion of low displacement rank. Our structured transforms admit fast function and gradient evaluation, and span a rich range of parameter sharing configurations whose statistical modeling capacity can be explicitly tuned along a continuum from structured to unstructured. Experimental results show that these transforms can significantly accelerate inference and forward/backward passes during training, and offer superior accuracy-compactness-speed tradeoffs in comparison to a number of existing techniques. In keyword spotting applications in mobile speech recognition, our methods are much more effective than standard linear low-rank bottleneck layers and nearly retain the performance of state of the art models, while providing more than 3.5-fold compression.

In this paper, the author proposes a method for fast matrix-vector multiplication and gradient computation for structured matrices. The notion of structured matrices is a generalized version of the Toeplitz matrix. Nevertheless, the technique of this paper can also be generalized to capture Vandermonde and Cauchy matrices.The main idea behind the methodology is to represent the matrix as a product of Z_1(G) and Z_{-1}(H), where both G and H are tall matrices. This is a generalization of the low-rank factorization. The author shows that this new representation not only captures the low-rank matrices, but also the product, inverse and linear combination of Toeplitz matrices. It was also shown that the matrix-vector product can be computed efficiently through a small number of FFTs, so that it is order-of-magnitude faster than dense matrix-vector multiplication.The idea is clever and is promising in accelerating the training of neural networks. It is also useful for compressing the representation of a neural network on mobile devices.Here are some comment:1. Toeplitz is closely related to one-dimensional convolution, but in practice use the convolution is usually two-dimensional. It means that using the Toeplitz representation may lose some information captured by the ordinary CNN. To see the impact of this loss, it is important to include ordinary CNN as a baseline algorithm, and compare algorithms on datasets other than MNIST. Although one dimensional convolution may be sufficient for MNIST classification. It may not be the case for natural images.2. It is somewhat surprising that when n = 512, the implementation proposed in this paper is almost surely slower than the dense matrix multiplication. The author may want to discuss a little why it happens. It seems that the BLAS implementation of dense matrix-vector multiplication is 100x faster for executing task of the same computation complexity?3. In mobile search recognition, what is the specific algorithm for training these models? Does the author train a fully connected network at first, then shrink it to be a smaller one, or does the author train the smaller model directly? If it was the first case, then what is the algorithm for shrinking the network?


Regret-Based Pruning in Extensive-Form Games
Counterfactual Regret Minimization (CFR) is a leading algorithm for finding a Nash equilibrium in large zero-sum imperfect-information games. CFR is an iterative algorithm that repeatedly traverses the game tree, updating regrets at each information set. We introduce an improvement to CFR that prunes any path of play in the tree, and its descendants, that has negative regret. It revisits that sequence at the earliest subsequent CFR iteration where the regret could have become positive, had that path been explored on every iteration. The new algorithm maintains CFR's convergence guarantees while making iterations significantly faster-even if previously known pruning techniques are used in the comparison. This improvement carries over to CFR+, a recent variant of CFR. Experiments show an order of magnitude speed improvement, and the relative speed improvement increases with the size of the game.

The paper describes a new way to prune parts of the game trees in counterfactual regret minimization. This is achieved by properties of regret-matching and by mixing best responses with CFR inspired by CFR-BR. The end result is by doing some extra book-keeping, much of the CFR tree pass can be omitted and then retroactively adapted at a later point when relevant.Regret-based pruning saves significant computation in terms of nodes touched per iteration and the savings seems to grow as a function of the game. The pruning technique is tied to CFR, but the authors also show improvements in CFR+.I have some questions to help clarify some points:Best response strategies normally prescribe an action (or distribution) for each one of a player's information sets, so the partial best responses in Section 3 are a not clear.Q1. Are these best responses to a well-defined subgame of Player 2? If so, how is the subgame defined?Q2. Assuming "no" to Q1, then the best responses under the zero-reach information sets are in fact just subsets of a full best response strategy. Can you explain why it's possible to directly re-use the CFR-BR theory in this case, where the strategy used from T_0 to T_0 + m is a combined (partial CFR strategy) and (partial best response)? This seems to assume that the games are decomposable.Q3. Can you give some kind of indication as to the raw computation time trade-off? How much overhead does the best response computation take? In the graphs, does "nodes touched" also somehow include the extra time taken for the best response calculations?---- Added after the review + discussion: The most efficient implementation of partial pruning for CFR can be found in code pointed to by "Limit Texas Hold'em is Solved" paper, reference [1]. Using this implementation, each iteration updates regrets for player i and updates the average strategy for player 3-i, alternating the players. This allows partial pruning when just one of the reaches (pi_{-i}) = 0. If accepted for publication, the authors should rephrase or soften their claim in the intro that CFR requires both reaches to 0 to apply partial pruning because, as admitted by the authors in their response, the validity of this claim is implementation-dependent. Specifically, the claim is not true for this most efficient implementation of Vanilla CFR, and thus this claim as written could mislead readers.Also, while it is not entirely clear, the benefit of RBP could be significantly affected if the authors had used this most efficient version of Vanilla CFR to compare to. Any future works/publication should either use this most efficient implementation as a baseline for comparison, or compare against both implementations, since the motivational arguments used in the intro do not hold for this implementation.


Unsupervised Learning by Program Synthesis
We introduce an unsupervised learning algorithm that combines probabilistic modeling with solver-based techniques for program synthesis. We apply our techniques to both a visual learning domain and a language learning problem, showing that our algorithm can learn many visual concepts from only a few examples and that it can recover some English inflectional morphology. Taken together, these results give both a new approach to unsupervised learning of symbolic compositional structures, and a technique for applying program synthesis tools to noisy data.

This paper works on the idea of explaining a set of data points x_i i = 1..N in terms of finding a program (drawn from a prior over programs, and common for all i = 1..N), and specific inputs I_i (i=1..N) for each datapoint. The program is built up out of domain specific primitives, as shown in Tables 1 and 2. The method is appliedto visual concept learning and morphological rule learning tasks which have a cognitive science flavor.Of course this idea is quite similar to standard latent variable modelling. However, rather than say having a matrix of parameters (asin factor analysis) or cluster centers (as in clustering) which define the model, we are allowed here a more general program to map fromI to x, see e.g. Fig 1 (right). However, the program can make use ofexplicit domain-specific prior knowledge, as per Tables 1 and 2.For sec 2 see comments underClarity.In sec 3.1 the method is applied to data from the SVRT (Synthetic Visual Reasoning Task), as per Fig 1. The evaluation as per line 203 is quite hard to understand, but I made sense of if as comparing P_x(t|E_j) for models based on different sets of examples j = 1,2. The paper has an inadequate discussion of the results -- based on raw accuracy (dashed lines in Fig 3) it seems that the program synthesis not doing much better than "image features" (0.78 vs 0.75)? But was the "image features" method trained with far more examples? The r values quoted assess comparison with human accuracy, but what is the goal of the paper -- unsupervised learning, or cognitive modeling?In sec 3.2 the method is applied to morphological rule learning. Again the details are very compressed -- words are represented by sequences of phonemes.I have no idea from the description how features like VOICED are defined -- e.g. do they apply to phonemes, or the word as a whole? Clarity: I found the paper rather lacking in detail -- this is likely due to there being a lot of ideas that are brought together. It is clearly hard to fit these into a NIPS-length paper, but it would have been possible to use the supp mat to expand on various issues and make them intelligible.For example Alg 1 (SMT encoding of expressions) is impenetrable. K(N', N'' ..) is undefined. This whole thing needs much more explanation; it remains very unclear to me what the basic idea of SMT solving is here and how it works. A worked example (e.g. in supp mat) would help a lot. Minor: What is the meaning of the if() function used in Alg 1?See also comments above about level of detail in ecs 3.1 and 3.2. Given these issues I suggest that the authors write a long paper which they make available as supp mat -- in this case the NIPS paper would really be a summary of that long paper.The statements about key ideas in l 422-424 could usefully be moved up to the introduction.Originality: Wrt the key ideas mentioned in l 422-424, the point about a domain specific grammar seems to be a common idea, e.g."background knowledge" in ILP. The "encoding of soft probabilisticconstraints as hard constraints for symbolic search" may be more novel, but the brevity of sec 2.2 makes it hard to understand this in detail.Significance: This paper is tackling a hard and interesting problem, that of the unsupervised learning of programs; this generalizes unsupervised learning applied to vector data to other kinds of data (e.g. sequences). Overall: Part of the difficulty in understanding this paper is that it is actually covering a lot of ground, and it is hard to fit this into the NIPS format. I have no knowledge about SMT, but if theauthors' claim about novelty wrt "encoding of soft probabilisticconstraints as hard constraints for symbolic search" is correct, then it seems to me that the paper is making a useful advance in a hard area. But it would be really helpful if the authors could producea significantly longer version where there is actually space toexplain what is going on.Minor:l 098 \Sigma -> \tau -> \tau ??N in Alg 1 contradicts its defn in l 077.Fig 3 -- the different colors are unnecessary, and the use ofyellow makes the ConvNet plot hard to read. l 337 I happen to know about RANSAC, but many readers won't. It would be useful to explain that RANSAC uses a minimal set of examples to learn a model, and then evaluate it on all of the available examples.ADDED AFTER REBUTTAL: =====================The other reviews and the author response have been read. I retain my score of 7, but strongly encourage the authors to makeextensive supplementary material available; at the moment the paper is more like an "advert" (Donoho's phrase), and we need much more detail if others are able to reproduce the work. In para 1 of therebuttal the authoros say "we could include [...] an example of an unsupervised program synthesis problem being converted into an SMTformula". The "could" here is not enough -- we need to see more detail.


Combinatorial Bandits Revisited
This paper investigates stochastic and adversarial combinatorial multi-armed bandit problems. In the stochastic setting under semi-bandit feedback, we derive a problem-specific regret lower bound, and discuss its scaling with the dimension of the decision space. We propose ESCB, an algorithm that efficiently exploits the structure of the problem and provide a finite-time analysis of its regret. ESCB has better performance guarantees than existing algorithms, and significantly outperforms these algorithms in practice. In the adversarial setting under bandit feedback, we propose COMBEXP, an algorithm with the same regret scaling as state-of-the-art algorithms, but with lower computational complexity for some combinatorial problems.

The contributions of this paper are threefold:1) For the stochastic combinatorial semi-bandits, the authors have derived a problem-specific lower bound and discussed its scaling.2) For the stochastic combinatorial semi-bandits, the authors have achieved an improved regret bound under some *additional assumptions*.3) For the adversarial combinatorial bandits, the authors have proposed an algorithm with the same regret scaling as the state of the arts, but with lower computational complexity for some problems.To the best of my knowledge, all these contributions are novel and nontrivial, though I have not seen a *fundamentally new* idea in this paper. The authors are quite familiar with the literature and have done a good literature review. I think the paper is ready to be published in NIPS after some appropriate revisions.My major comment is that the authors should position the paper better with respect to some existing literature. In particular, with respect to Kveton et al. 2015 [11], contribution 2) of this paper has achieved better regret bound under some *additional assumptions* (i.e. this paper assumes the rewards are Bernoulli and independent across actions, while [11] does not make these assumptions). The authors should clarify these differences in the next version of this paper. Moreover, since [11] claims that their regret bound is tight, to avoid confusion, the authors should explicitly mention that the particular example [11] used to derive their lower bound does not satisfy the assumptions of this paper.There are some recent papers on combinatorial bandits/semi-bandits, and the authors might consider citing them:http://jmlr.org/proceedings/papers/v40/Neu15.html http://jmlr.org/proceedings/papers/v37/wen15Some minor comments:1) Line 222, \Delta^M should be \Delta_M;2) In equation 2, both M and t(n) are vectors, so Mt(n) is not well defined.


Semi-Supervised Learning with Ladder Networks
We combine supervised learning with unsupervised learning in deep neural networks. The proposed model is trained to simultaneously minimize the sum of supervised and unsupervised cost functions by backpropagation, avoiding the need for layer-wise pre-training. Our work builds on top of the Ladder network proposed by Valpola [1] which we extend by combining the model with supervision. We show that the resulting model reaches state-of-the-art performance in semi-supervised MNIST and CIFAR-10 classification in addition to permutationinvariant MNIST classification with all labels.

The manuscript presents impressive semi-supervised results for MNIST using a simple idea related to denoising autoencoders.The paper proposes a layer-wise reconstruction error of pre-noised activations using a function that is dependent on top down connections and knowledge of the noisy activations at a given layer, i.e. \hat{z}^{l}= g(\noise{z}^{l},\noise{z}^{l+t}).The approach straightforwardly combines this errors into a scalar loss, combines this unsupervised loss with the supervised loss and trains end-to-end.the manuscript also presents good, though slightly less striking, results for the CIFAR dataset using the same approach.It would be nice to see the approach applied to one additional dataset (e.g. demonstrating that it is practical on at least a subset of ImageNet), but I don't believe this is important for acceptance.While the technique is simple the introduction makes it more difficult to follow than need be.The didactic Figure 1 (left panel) in particular could be improved for legibility.Could the choice of the g() function in equation (1) be better justified?Likely I'm missing something obvious, but why does the vector v have duplicate elements (line 190)?It may be helpful to other readers to make this clear.There are also some small miscellaneous grammatical and spelling issues throughout.The title of the paper might might work better if the work network was pluralized: e.g., "Semi-supervised learning with ladder networks".When comparing to past results, it should be made clear which past papers made an effort to "balance classes" for the small subsets used for training.Balancing classes almost certainly helps performance, so it would be good to note which past results had the benefit of balanced classes and which did not.As a control it would also be nice to compare the results presented with a more "vanilla" semi-supervised denoising autoencoder approach using the same architecture (e.g. batch normalization and relu activations).This would give us an idea about how much of the performance gain is due to the new ladder-network idea versus just using slightly more updated training methods.Finally, though the computational costs of the semi-supervised approach seem quite reasonable, but it would be nice to have some measure of the expected wall-clock time/performance for a well optimized version of the ladder network approach versus the purely supervised benchmark.Demonstration that this kind of semi-supervised approach only requires a modest amount of extra wall-time may help foster wider adoption of this kind of mechanism and help us push the through the tendency to use purely-supervised that currently dominates the field.


The Population Posterior and Bayesian Modeling on Streams
Many modern data analysis problems involve inferences from streaming data. However, streaming data is not easily amenable to the standard probabilistic modeling approaches, which require conditioning on finite data. We develop population variational Bayes, a new approach for using Bayesian modeling to analyze streams of data. It approximates a new type of distribution, the population posterior, which combines the notion of a population distribution of the data with Bayesian inference in a probabilistic model. We develop the population posterior for latent Dirichlet allocation and Dirichlet process mixtures. We study our method with several large-scale data sets.

- Summary of Paper   - The paper describes the development of an evidence lower bound     (ELBO) constructed by averaging over a data-generating     distribution. The paper shows that optimizing this ELBO leads to     impressive results on very large data streaming applications. - Quality   - L039: "the standard approach to probabilitic modeling", might be     better stated as "the standard approach to Bayesian probabilitic     modeling" since we can build a probabilitic model without a data     set in hand.   - L044: On initial reading, my reaction was, "Why shouldn't the     Bayesian posterior become more confident with more data? Indeed,     this is a desirable property of Bayesian inference procedures.     Also, "over-dispersion" is a well-known problem for many     generalized linear models and the typical solution there is to     build a better model. So, isn't the solution here, to build a     better model? Or if there is uncertainty about the model, perhaps     we should average over models in some way." However, later, some     clarity is provided in that the procedure aims to be robust to     model specification in a different way.   - L051: Again, my initial reaidng of this paragraph caused me to be     concerned that the real problem with Bayesian updating on data     streams is not the Bayesian procedure, but the way the model has     been specified. If the data stream is changing and we haven't     explicitely modeled that, then of course the updates may yeild     poor inferences, but that's not because our updating procedure is     flawed, but because our model is flawed. Here again, it seems     that the proposed procedure is trying to be robust to model     specification issues that really cause problem on data streams.     Perhaps the narrative in these introductory paragraphs can be     sharpened to set up the nice work presented later.   - L056: The claim is that explicitly modeling the time series     incurs a heavy inferential cost. Can this claim be supported with     a citation or other evidence?   - L165: Is there a misplaced parenthesis and perhaps a missing     \beta in the variantional distribution in the F-ELBO?   - L165: The standard ELBO is conditional on a particular data set x     and the F-ELBO is an average over data set x provided by the     population distribution X ~ F_\alpha. I'm curious if taking this     average causes the F-ELBO to preferentially optimize the ELBO     over modes of F_\alpha. Whereas, if we conditioned on a     particular x, as in the ELBO, it wouldn't matter how likely that     data set is under F_\alpha. Can the authors comment on the     tradeoffs of marginalizing over F_\alpha versus conditioning on a     sample from it?   - The results primarily deal with prediction rather than parameter     estimation. This is entirely appropriate given the applications     where streaming data is typically found. However, is there     anything that can be said about the parameter estimates,     especially given the first-order goal of maximizing the ELBO or     F-ELBO is to obtain parameter estimates?   - I do like that the F-ELBO explains SVI and provides a nice     framework for understanding that sampling procedure. But, I     wonder if one has in hand a generative model for p(X), what is     the costs/benefits of using that distribution as an averaging     distribution instead of X ~ F_\alpha? I understand that if our     model is misspecified, averaging with respect to that model may     exacerbate the updating problems outlined, and instead drawing     samples from F_\alpha is model-independent. Is there any guidance     as to another reason p(X) is a poor choice? - Clarity   - It would help to clarify exactly where the problems identified in     paragraph 2 and 3 in the introduction lie. L042 says that the     problems are with "Bayesian updating on data streams", but L044     says "the first problem is that Bayesian posteriors will become     overconfident" and L051 says "the data stream might change over     time". After reading these assertions several times, it becomes     clear what is intended, but I think the statement on L044 could     be better as "The first problem with Bayesian updating on data     streams is that Bayesian posteriors will become overconfident"     and L051 could be "The second problem with Bayesian updating on     data streams is that the data stream might change over time     causing ..." - Originality   - The paper is original and provides a good justification for SVI. - Significance   - I find the paper to be highly significant and I hope will be a     welcome addition in the community.


An efficient learner is one who reuses what they already know to tackle a new problem. For a machine learner, this means understanding the similarities amongst datasets. In order to do this, one must take seriously the idea of working with datasets, rather than datapoints, as the key objects to model. Towards this goal, we demonstrate an extension of a variational autoencoder that can learn a method for computing representations, or statistics, of datasets in an unsupervised fashion. The network is trained to produce statistics that encapsulate a generative model for each dataset. Hence the network enables efficient learning from new datasets for both unsupervised and supervised tasks. We show that we are able to learn statistics that can be used for: clustering datasets, transferring generative models to new datasets, selecting representative samples of datasets and classifying previously unseen classes. We refer to our model as a neural statistician, and by this we mean a neural network that can learn to compute summary statistics of datasets without supervision.
1 INTRODUCTION
The machine learning community is well-practised at learning representations of data-points and sequences. A middle-ground between these two is representing, or summarizing, datasets - unordered collections of vectors, such as photos of a particular person, recordings of a given speaker or a document as a bag-of-words. Where these sets take the form of i.i.d samples from some distribution, such summaries are called statistics. We explore the idea of using neural networks to learn statistics and we refer to our approach as a neural statistician.
The key result of our approach is a statistic network that takes as input a set of vectors and outputs a vector of summary statistics specifying a generative model of that set - a mean and variance specifying a Gaussian distribution in a latent space we term the context. The advantages of our approach are that it is:
• Unsupervised: It provides principled and unsupervised way to learn summary statistics as the output of a variational encoder of a generative model.
• Data efficient: If one has a large number of small but related datasets, modelling the datasets jointly enables us to gain statistical strength.
• Parameter Efficient: By using summary statistics instead of say categorical labellings of each dataset, we decouple the number of parameters of the model from the number of datasets.
• Capable of few-shot learning: If the datasets correspond to examples from different classes, class embeddings (summary statistics associated with examples from a class), allow us to handle new classes at test time.
2 PROBLEM STATEMENT
We are given datasets Di for i ∈ I. Each dataset Di = {x1, . . . , xki} consists of a number of i.i.d samples from an associated distribution pi over Rn. The task can be split into learning and inference components. The learning component is to produce a generative model p̂i for each dataset Di. We assume there is a common underlying generative process p such that pi = p(·|ci) for ci ∈ Rl drawn
from p(c). We refer to c as the context. The inference component is to give an approximate posterior over the context q(c|D) for a given dataset produced by a statistic network.
3 NEURAL STATISTICIAN
In order to exploit the assumption of a hierarchical generative process over datasets we will use a ‘parameter-transfer approach’ (see Pan & Yang, 2010) to extend the variational autoencoder model of Kingma & Welling (2013).
3.1 VARIATIONAL AUTOENCODER
The variational autoencoder is a latent variable model p(x|z; θ) (often called the decoder) with parameters θ. For each observed x, a corresponding latent variable z is drawn from p(z) so that
p(x) = ∫ p(x|z; θ)p(z) dz. (1)
The generative parameters θ are learned by introducing a recognition network (also called an encoder) q(z|x;φ) with parameters φ. The recognition network gives an approximate posterior over the latent variables that can then be used to give the standard variational lower bound (Saul & Jordan, 1996) on the single-datum log-likelihood. I.e. logP (x|θ) ≥ Lx, where
Lx = Eq(z|x,φ) [log p(x|z; θ)]−DKL (q(z|x;φ)‖p(z)) . (2)
Likewise the full-data log likelihood is lower bounded by the sum of the Lx terms over the whole dataset. We can then optimize this lower bound with respect to φ and θ using the reparameterization trick introduced by Kingma & Welling (2013) and Rezende et al. (2014) to get a Monte-Carlo estimate of the gradient.
3.2 BASIC MODEL
We extend the variational autoencoder to the model depicted on the left in Figure 1. This includes a latent variable c, the context, that varies between different datasets but is constant, a priori, for items within the same dataset. Now, the likelihood of the parameters θ for one single particular dataset D is given by
p(D) = ∫ p(c) [∏ x∈D ∫ p(x|z; θ)p(z|c; θ) dz ] dc. (3)
The prior p(c) is chosen to be a spherical Gaussian with zero mean and unit variance. The conditional p(z|c; θ) is Gaussian with diagonal covariance, where all the mean and variance parameters depend on c through a neural network. Similarly the observation model p(x|z; θ) will be a simple likelihood function appropriate to the data modality with dependence on z parameterized by a neural network. For example, with real valued data, a diagonal Gaussian likelihood could be used where the mean and log variance of x are created from z via a neural network.
We use approximate inference networks q(z|x, c;φ), q(c|D;φ), with parameters collected into φ, to once again enable the calculation and optimization of a variational lower bound on the loglikelihood. The single dataset log likelihood lower bound is given by
LD = Eq(c|D;φ) [∑ x∈d Eq(z|c,x;φ) [log p(x|z; θ)]−DKL (q(z|c, x;φ)‖p(z|c; θ)) ] −DKL (q(c|D;φ)‖p(c)) . (4)
As with the generative distributions, the likelihood forms for q(z|x, c;φ) and q(c|D;φ) are diagonal Gaussian distributions, where all the mean and log variance parameters in each distribution are produced by a neural network taking the conditioning variables as inputs. Note that q(c|D;φ) accepts as input a dataset D and we refer to this as the statistic network. We describe this in Subsection 3.4.
The full-data variational bound is given by summing the variational bound for each dataset in our collection of datasets. It is by learning the difference of the within-dataset and between-dataset distributions that we are able to discover an appropriate statistic network.
3.3 FULL MODEL
The basic model works well for modelling simple datasets, but struggles when the datasets have complex internal structure. To increase the sophistication of the model we use multiple stochastic layers z1, . . . , zk and introduce skip-connections for both the inference and generative networks. The generative model is shown graphically in Figure 1 in the center. The probability of a dataset D is then given by
p(D) = ∫ p(c) ∏ x∈D ∫ p(x|c, z1:L; θ)p(zL|c; θ) L−1∏ i=1 p(zi|zi+1, c; θ) dz1:L dc (5)
where the p(zi|zi+1, c, θ) are again Gaussian distributions where the mean and log variance are given as the output of neural networks. The generative process for the full model is described in Algorithm 1.
The full approximate posterior factorizes analogously as
q(c, z1:L|D;φ) = q(c|D;φ) ∏ x∈D q(zL|x, c;φ) L−1∏ i=1 q(zi|zi+1, x, c;φ). (6)
For convenience we give the variational lower bound as sum of a three parts, a reconstruction term RD, a context divergence CD and a latent divergence LD:
LD = RD + CD + LD with (7) RD = Eq(c|D;φ) ∑ x∈D Eq(z1:L|c,x;φ) log p(x|z1:L, c; θ) (8)
CD = DKL (q(c|D;φ)‖p(c)) (9)
LD = Eq(c,z1:L|D;φ) [∑ x∈D DKL (q(zL|c, x;φ)‖p(zL|c; θ))
+ L−1∑ i=1 DKL (q(zi|zi+1, c, x;φ)‖p(zi|zi+1, c; θ))
] . (10)
The skip-connections p(zi|zi+1, c; θ) and q(zi|zi+1, x;φ) allow the context to specify a more precise distribution for each latent variable by explaining-away more generic aspects of the dataset at each stochastic layer. This architecture was inspired by recent work on probabilistic ladder networks in Kaae Sønderby et al. (2016). Complementing these are the skip-connections from each latent variable to the observation p(x|z1:L, c; θ), the intuition here is that each stochastic layer can focus on representing a certain level of abstraction, since its information does not need to be copied into the next layer, a similar approach was used in Maaløe et al. (2016).
Once again, note that we are maximizing the lower bound to the log likelihood over many datasets D: we want to maximize the expectation of LD over all datasets. We do this optimization using stochastic gradient descent. In contrast to a variational autoencoder where a minibatch would consist of a subsample of datapoints from the dataset, we use minibatches consisting of a subsample of datasets - tensors of shape (batch size, sample size, number of features).
3.4 STATISTIC NETWORK
In addition to the standard inference networks we require a statistic network q(c|D;φ) to give an approximate posterior over the context c given a datasetD = {x1, . . . , xk} . This inference network must capture the exchangeability of the data in D.
We use a feedforward neural network consisting of three main elements:
• An instance encoder E that takes each individual datapoint xi to a vector ei = E(xi). • An exchangeable instance pooling layer that collapses the matrix (e1, . . . , ek) to a single
pre-statistic vector v. Examples include elementwise means, sums, products, geometric means and maximum. We use the sample mean for all experiments. • A final post-pooling network that takes v to a parameterization of a diagonal Gaussian.
The graphical model for this is given at the right of Figure 1.
We note that the humble sample mean already gives the statistic network a great deal of representational power due to the fact that the instance encoder can learn a representation where averaging makes sense. For example since the instance encoder can approximate a polynomial on a compact domain, and so can the post-pooling network, a statistic network can approximate any moment of a distribution.
4 RELATED WORK
Due to the general nature of the problem considered, our work touches on many different topics which we now attempt to summarize.
Topic models and graphical models The form of the graphical model in Figure 1 on the left is equivalent to that of a standard topic model. In contrast to traditional topic models we do not use discrete latent variables, or restrict to discrete data. Work such as that by Ranganath et al. (2014) has extended topic models in various directions, but importantly we use flexible conditional distributions and dependency structures parameterized by deep neural networks. Recent work has explored neural networks for document models (see e.g. Miao et al., 2015) but has been limited to modelling datapoints with little internal structure. Along related lines are ‘structured variational autoencoders’ (see Johnson et al., 2016), where they treat the general problem of integrating graphical models with variational autoencoders.
Transfer learning There is a considerable literature on transfer learning, for a survey see Pan & Yang (2010). There they discuss ‘parameter-transfer’ approaches whereby parameters or priors are shared across datasets, and our work fits into that paradigm. For examples see Lawrence & Platt (2004) where share they priors between Gaussian processes, and Evgeniou & Pontil (2004) where they take an SVM-like approach to share kernels.
One-shot Learning Learning quickly from small amounts of data is a topic of great interest. Lake et al. (2015) use Bayesian program induction for one-shot generation and classification, and Koch (2015) train a Siamese (Chopra et al. (2005)) convolutional network for one-shot image classification. We note the relation to the recent work (Rezende et al., 2016) in which the authors use a conditional recurrent variational autoencoder capable of one-shot generalization by taking as extra input a conditioning data point. The important differences here are that we jointly model datasets and datapoints and consider datasets of any size. Recent approaches to one-shot classification are matching networks (Vinyals et al., 2016b) (which was concurrent with the initial preprint of this work), and related previous work (Santoro et al., 2016). The former can be considered a kind of differentiable nearest neighbour classifier, and the latter augments their network with memory to store information about the classification problem. Both are trained end-to-end for the classification problem, whereas the present work is a general approach to learning representations of datasets. Probably the closest previous work is by Salakhutdinov et al. (2012) where the authors learn a topic
model over the activations of a DBM for one-shot learning. Compared with their work we use modern architectures and easier to train VAEs, in particular we have fast and amortized feedforward inference for test (and training) datasets, avoiding the need for MCMC.
Multiple-Instance Learning There is previous work on classifying sets in multiple-instance learning, for a useful survey see Cheplygina et al. (2015). Typical approaches involve adapting kernel based methods such as support measure machines (Muandet et al., 2012), support distribution machines (Póczos et al., 2012) and multiple-instance-kernels (Gartner et al., 2002). We do not consider applications to multiple-instance learning type problems here, but it may be fruitful to do so in the future.
Set2Seq In very related work, Vinyals et al. (2016a) explore architectures for mapping sets to sequences. There they use an LSTM to repeatedly compute weighted-averages of the datapoints and use this to tackle problems such as sorting a list of numbers. The main difference between their work and ours is that they primarily consider supervised problems, whereas we present a general unsupervised method for learning representations of sets of i.i.d instances. In future work we may also explore recurrently computing statistics.
ABC There has also been work on learning summary statistics for Approximate Bayesian Computation by either learning to predict the parameters generating a sample as a supervised problem, or by using kernel embeddings as infinite dimensional summary statistics. See the work by Fukumizu et al. (2013) for an example of kernel-based approaches. More recently Jiang et al. (2015) used deep neural networks to predict the parameters generating the data. The crucial differences are that their problem is supervised, they do not leverage any exchangeability properties the data may have, nor can it deal with varying sample sizes.
5 EXPERIMENTAL RESULTS
Given an input set x1, . . . xk we can use the statistic network to calculate an approximate posterior over contexts q(c|x1, . . . , xk;φ). Under the generative model, each context c specifies a conditional model p(x|c; θ). To get samples from the model corresponding to the most likely posterior value of c, we set c to the mean of the approximate posterior and then sample directly from the conditional distributions. This is described in Algorithm 2. We use this process in our experiments to show samples. In all experiments, we use the Adam optimization algorithm (Kingma & Ba, 2014) to optimize the parameters of the generative models and variational approximations. Batch normalization (Ioffe & Szegedy, 2015) is implemented for convolutional layers and we always use a batch size of 16. We primarily use the Theano (Theano Development Team, 2016) framework with the Lasagne (Dieleman et al., 2015) library, but the final experiments with face data were done using Tensorflow (Abadi et al., 2015). In all cases experiments were terminated after a given number of epochs when training appeared to have sufficiently converged (300 epochs for omniglot, youtube and spatial MNIST examples, and 50 epochs for the synthetic experiment).
5.1 SIMPLE 1-D DISTRIBUTIONS
In our first experiment we wanted to know if the neural statistician will learn to cluster synthetic 1-D datasets by distribution family. We generated a collection of synthetic 1-D datasets each containing 200 samples. Datasets consist of samples from either an Exponential, Gaussian, Uniform or Laplacian distribution with equal probability. Means and variances are sampled from U [−1, 1] and U [0.5, 2] respectively. The training data contains 10K sets.
The architecture for this experiment contains a single stochastic layer with 32 units for z and 3 units for c, . The model p(x|z, c; θ) and variational approximation q(z|x, c;φ) are each a diagonal Gaussian distribution with all mean and log variance parameters given by a network composed of three dense layers with ReLU activations and 128 units. The statistic network determining the mean and log variance parameters of posterior over context variables is composed of three dense layers before and after pooling, each with 128 units with Rectified Linear Unit (ReLU) activations.
Figure 2 shows 3-D scatter plots of the summary statistics learned. Notice that the different families of distribution cluster. It is interesting to observe that the Exponential cluster is differently orientated to the others, perhaps reflecting the fact that it is the only non-symmetric distribution. We also see that between the Gaussian and Laplacian clusters there is an area of ambiguity which is as one
might expect. We also see that within each cluster the mean and variance are mapped to orthogonal directions.
5.2 SPATIAL MNIST
Building on the previous experiments we investigate 2-D datasets that have complex structure, but the datapoints contain little information by themselves, making it a good test of the statistic network. We created a dataset called spatial MNIST. In spatial MNIST each image from MNIST (LeCun et al., 1998) is turned into a dataset by interpreting the normalized pixel intensities as a probability density and sampling coordinate values. An example is shown in Figure 3. This creates two-dimensional spatial datasets. We used a sample size of 50. Note that since the pixel coordinates are discrete, it is necessary to dequantize them by adding uniform noise u ∼ U [0, 1] to the coordinates if one models them as real numbers, else you can get arbitrarily high densities (see Theis et al. (2016) for a discussion of this point).
The generative architecture for this experiment contains 3 stochastic z layers, each with 2 units, and a single c layer with 64 units. The means and log variances of the Gaussian likelihood for p(x|z1:3, c; θ), and each subnetwork for z in both the encoder and decoder contained 3 dense layers with 256 ReLU units each. The statistic network also contained 3 dense layers pre-pooling and 3 dense layers post pooling with 256 ReLU units.
In addition to being able to sample from the model conditioned on a set of inputs, we can also summarize a dataset by choosing a subset S ⊆ D to minimise the KL divergence of q(C|D;φ) from q(C|S;φ). We do this greedily by iteratively discarding points from the full sample. Pseudocode for this process is given in Algorithm 3. The results are shown in Figure 4. We see that the model is capable of handling complex arrangements of datapoints. We also see that it can select sensible subsets of a dataset as a summary.
5.3 OMNIGLOT
Next we work with the OMNIGLOT data (Lake et al., 2015). This contains 1628 classes of handwritten characters but with just 20 examples per class. This makes it an excellent test-bed for transfer / few-shot learning. We constructed datasets by splitting each class into datasets of size 5. We train
on datasets drawn from 1200 classes and reserve the remaining classes to test few-shot sampling and classification. We created new classes by rotating and reflecting characters. We resized the images to 28 × 28. We sampled a binarization of each image for each epoch. We also randomly applied the dilation operator from computer vision as further data augmentation since we observed that the stroke widths are quite uniform in the OMNIGLOT data, whereas there is substantial variation in MNIST, this augmentation improved the visual quality of the few-shot MNIST samples considerably and increased the few-shot classification accuracy by about 3 percent. Finally we used ‘sample dropout’ whereby a random subset of each dataset was removed from the pooling in the statistic network, and then included the number of samples remaining as an extra feature. This was beneficial since it reduced overfitting and also allowed the statistic network to learn to adjust the approximate posterior over c based on the number of samples.
We used a single stochastic layer with 16 units for z, and 512 units for c. We used a shared convolutional encoder between the inference and statistic networks and a deconvolutional decoder network. Full details of the networks are given in Appendix B.1. The decoder used a Bernoulli likelihood.
In Figure 5 we show two examples of few-shot learning by conditioning on samples of unseen characters from OMNIGLOT, and conditioning on samples of digits from MNIST. The samples are mostly of a high-quality, and this shows that the neural statistician can generalize even to new datasets.
As a further test we considered few-shot classification of both unseen OMNIGLOT characters and MNIST digits. Given a sets of labelled examples of each class D0, . . . , D9 (for MNIST say), we computed the approximate posteriors q(C|Di;φ) using the statistic network. Then for each test image x we also computed the posterior q(C|x;φ) and classified it according to the training dataset Di minimizing the KL divergence from the test context to the training context. This process is described in Algorithm 4. We tried this with either 1 or 5 labelled examples per class and either 5 or 20 classes. For each trial we randomly select K classes, randomly select training examples for each class, and test on the remaining examples. This process is repeated 100 times and the results averaged. The results are shown in Table 1. We compare to a number of results reported in Vinyals et al. (2016b) including Santoro et al. (2016) and Koch (2015). Overall we see that
the neural statistician model can be used as a strong classifier, particularly for the 5-way tasks, but performs worse than matching networks for the 20-way tasks. One important advantage that matching networks have is that, whilst each class is processed independently in our model, the representation in matching networks is conditioned on all of the classes in the few-shot problem. This means that it can exaggerate differences between similar classes, which are more likely to appear in a 20-way problem than a 5-way problem.
5.4 YOUTUBE FACES
Finally, we provide a proof of concept for generating faces of a particular person. We use the Youtube Faces Database from Wolf et al. (2011). It contains 3, 245 videos of 1, 595 different people. We use the aligned and cropped to face version, resized to 64 × 64. The validation and test sets contain 100 unique people each, and there is no overlap of persons between data splits. The sets were created by sampling frames randomly without replacement from each video, we use a set size of 5 frames. We resample the sets for the training data each epoch.
Our architecture for this problem is based on one presented in Lamb et al. (2016). We used a single stochastic layer with 500 dimensional latent c and 16 dimensional z variable. The statistic network and the inference network q(z|x, c;φ) share a common convolutional encoder, and the deocder uses deconvolutional layers. For full details see Appendix B.2. The likelihood function is a Gaussian, but where the variance parameters are shared across all datapoints, this was found to make training faster and more stable.
The results are shown in Figure 6. Whilst there is room for improvement, we see that it is possible to specify a complex distribution on-the-fly with a set of photos of a previously unseen person. The samples conditioned on an input set have a reasonable likeness of the input faces. We also show the ability of the model to generate new datasets and see that the samples have a consistent identity and varied poses.
6 CONCLUSION
We have demonstrated a highly flexible model on a variety of tasks. Going forward our approach will naturally benefit from advances in generative models as we can simply upgrade our base generative model, and so future work will pursue this. Compared with some other approaches in the literature for few-shot learning, our requirement for supervision is weaker: we only ask at training time that we are given datasets, but we do not need labels for the datasets, nor even information on whether two datasets represent the same or different classes. It would be interesting then to explore application areas where only this weaker form of supervision is available. There are two important limitations to this work, firstly that the method is dataset hungry: it will likely not learn useful representations of datasets given only a small number of them. Secondly at test time the few-shot fit of the generative model will not be greatly improved by using larger datasets unless the model was also trained on similarly large datasets. The latter limitation seems like a promising future research direction - bridging the gap between fast adaptation and slow training.
ACKNOWLEDGMENTS
This work was supported in part by the EPSRC Centre for Doctoral Training in Data Science, funded by the UK Engineering and Physical Sciences Research Council (grant EP/L016427/1) and the University of Edinburgh.
A APPENDIX A: PSEUDOCODE
Algorithm 1 Sampling a dataset of size k sample c ∼ p(c) for i = 1 to k do
sample zi,L ∼ p(zL|c; θ) for j = L− 1 to 1 do
sample zi,j ∼ p(zj |zi,j+1, c; θ) end for sample xi ∼ p(x|zi,1, . . . , zi,L, c; θ)
end for
Algorithm 2 Sampling a dataset of size k conditioned on a dataset of size m µc, σ 2 c ← q(c|x1, . . . , xm;φ) {Calculate approximate posterior over c using statistic network.}
c← µc {Set c to be the mean of the approximate posterior.} for i = 1 to k do
sample zi,L ∼ p(zL|c; θ) for j = L− 1 to 1 do
sample zi,j ∼ p(zj |zi,j+1, c; θ) end for sample xi ∼ p(x|zi,1, . . . , zi,L, c; θ)
end for
Algorithm 3 Selecting a representative sample of size k S ← {x1, . . . , xm} I ← {1, . . . ,m} SI = {xi ∈ S : i ∈ I} NSI ← q(c|SI ;φ) {Calculate approximate posterior over c using statistic network.} for i = 1 to k do t← argminj∈IDKL ( NS‖NSI−j
) I ← I − t
end for
Algorithm 4 K-way few-shot classification D0, . . . , DK ← sets of labelled examples for each class x← datapoint to be classified Nx ← q(c|x;φ) {approximate posterior over c given query point} for i = 1 to K do Ni ← q(c|Di;φ)
end for ŷ ← argminiDKL (Ni‖Nx)
B APPENDIX B: FURTHER EXPERIMENTAL DETAILS
B.1 OMNIGLOT
Shared encoder x→ h 2× { conv2d 64 feature maps with 3× 3 kernels and ELU activations } conv2d 64 feature maps with 3× 3 kernels, stride 2 and ELU activations 2× {conv2d 128 feature maps with 3× 3 kernels and ELU activations } conv2d 128 feature maps with 3× 3 kernels, stride 2 and ELU activations 2× { conv2d 256 feature maps with 3× 3 kernels and ELU activations } conv2d 256 feature maps with 3× 3 kernels, stride 2 and ELU activations
Statistic network q(c|D;φ) : h1, . . . , hk → µc, σ2c fully-connected layer with 256 units and ELU activations sample-dropout and concatenation with number of samples average pooling within each dataset 2× {fully-connected layer with 256 units and ELU activations } fully-connected linear layers to µc and log σ2c
Inference network q(z|x, c;φ) : h, c→ µz, σ2z concatenate c and h 3× {fully-connected layer with 256 units and ELU activations } fully-connected linear layers to µz and log σ2z
Latent decoder network p(z|c; θ) : c→ µz, σ2z 3× {fully-connected layer with 256 units and ELU activations } fully-connected linear layers to µz and log σ2z
Observation decoder network p(x|c, z; θ) : c, z → µx concatenate z and c fully-connected linear layers with 4 · 4 · 256 units 2× { conv2d 256 feature maps with 3× 3 kernels and ELU activations } deconv2d 256 feature maps with 2× 2 kernels, stride 2, ELU activations 2× { conv2d 128 feature maps with 3× 3 kernels and ELU activations } deconv2d 128 feature maps with 2× 2 kernels, stride 2, ELU activations 2× { conv2d 64 feature maps with 3× 3 kernels and ELU activations } deconv2d 64 feature maps with 2× 2 kernels, stride 2, ELU activations conv2d 1 feature map with 1× 1 kernels, sigmoid activations
B.2 YOUTUBE FACES
Shared encoder x→ h 2× { conv2d 32 feature maps with 3× 3 kernels and ELU activations } conv2d 32 feature maps with 3× 3 kernels, stride 2 and ELU activations 2× {conv2d 64 feature maps with 3× 3 kernels and ELU activations } conv2d 64 feature maps with 3× 3 kernels, stride 2 and ELU activations 2× { conv2d 128 feature maps with 3× 3 kernels and ELU activations } conv2d 128 feature maps with 3× 3 kernels, stride 2 and ELU activations 2× { conv2d 256 feature maps with 3× 3 kernels and ELU activations } conv2d 256 feature maps with 3× 3 kernels, stride 2 and ELU activations
Statistic network q(c|D,φ) : h1, . . . , hk → µc, σ2c fully-connected layer with 1000 units and ELU activations average pooling within each dataset fully-connected linear layers to µc and log σ2c
Inference network q(z|x, c, φ) : h, c→ µz, σ2z concatenate c and h fully-connected layer with 1000 units and ELU activations fully-connected linear layers to µz and log σ2z
Latent decoder network p(z|c, ; θ) : c→ µz, σ2z fully-connected layer with 1000 units and ELU activations fully-connected linear layers to µz and log σ2z
Observation decoder network p(x|c, z; θ) : c, z → µx concatenate z and c fully-connected layer with 1000 units and ELU activations fully-connected linear layer with 8 · 8 · 256 units 2× { conv2d 256 feature maps with 3× 3 kernels and ELU activations } deconv2d 256 feature maps with 2× 2 kernels, stride 2, ELU activations 2× { conv2d 128 feature maps with 3× 3 kernels and ELU activations } deconv2d 128 feature maps with 2× 2 kernels, stride 2, ELU activations 2× { conv2d 64 feature maps with 3× 3 kernels and ELU activations } deconv2d 64 feature maps with 2× 2 kernels, stride 2, ELU activations 2× { conv2d 32 feature maps with 3× 3 kernels and ELU activations } deconv2d 32 feature maps with 2× 2 kernels, stride 2, ELU activations conv2d 3 feature maps with 1× 1 kernels, sigmoid activations

The authors introduce a variant of the variational autoencoder (VAE) that models dataset-level latent variables. The idea is clearly motivated and well described. In my mind the greatest contribution of this paper is the movement beyond the relatively simple graphical model structure of the traditional VAEs and the introduction of more interesting structures to the deep learning community.
Comments:
- It's not clear to me why this should be called a "statistician". Learning an approximate posterior over summary statistics is not the only imaginable way to summarize a dataset with a neural network. One could consider a maximum likelihood approach, etc. In general it felt like the paper could be more clear, if it avoided coining new terms like "statistic network" and stuck to the more accurate "approximate posterior".
- The experiments are nice, and I appreciate the response to my question regarding "one shot generation". I still think that language needs to be clarified, specifically at the end of page 6. My understanding of Figure 5 is the following: Take an input set, compute the approximate posterior over the context vector, then generate from the forward model given samples from the approximate posterior. I would like clarification on the following:
(a) Are the data point dependent vectors z generated from the forward model or taken from the approximate posterior?
(b) I agree that the samples are of high-quality, but that is not a quantified statement. The advantage of VAEs over GANs is that we have natural ways of computing log-probabilities. To that end, one "proper" way of computing the "one shot generation" performance is to report log p(x | c) (where c is sampled from the approximate posterior) or log p(x) for held-out datasets. I suspect that log probability performance of these networks relative to a vanilla VAE without the context latent variable will be impressive. I still don't see a reason not to include that.