GENERATIVE MATCHING NETWORKS
Despite recent advances, the remaining bottlenecks in deep generative models are necessity of extensive training and difficulties with generalization from small number of training examples. Both problems may be addressed by conditional generative models that are trained to adapt the generative distribution to additional input data. So far this idea was explored only under certain limitations such as restricting the input data to be a single object or multiple objects representing the same concept. In this work we develop a new class of deep generative model called generative matching networks which is inspired by the recently proposed matching networks for one-shot learning in discriminative tasks and the ideas from meta-learning. By conditioning on the additional input dataset, generative matching networks may instantly learn new concepts that were not available during the training but conform to a similar generative process, without explicit limitations on the number of additional input objects or the number of concepts they represent. Our experiments on the Omniglot dataset demonstrate that generative matching networks can significantly improve predictive performance on the fly as more additional data is available to the model and also adapt the latent space which is beneficial in the context of feature extraction.
1 INTRODUCTION
Deep generative models are currently one of the most promising directions in generative modelling. In this class of models the generative process is defined by a composition of conditional distributions modelled using deep neural networks which form a hierarchy of latent and observed variables. This approach allows to build models with complex, non-linear dependencies between variables and efficiently learn the variability across training examples.
Such models are trained by stochastic gradient methods which can handle large datasets and a wide variety of model architectures but also present certain limitations. The training process usually consists of small, incremental updates of networks’ parameters and requires many passes over training data. Notably, once a model is trained it cannot be adapted to newly available data without complete re-training to avoid catastrophic interference (McCloskey & Cohen, 1989; Ratcliff, 1990). There is also a risk of overfitting for concepts that are not represented by enough training examples which is caused by high capacity of the models. Hence, most of deep generative models are not well-suited for rapid learning in one-shot scenario which is often encountered in real-world applications where data acquisition is expensive or fast adaptation to new data is required.
A potential solution to these problems is explicit learning of adaptation mechanisms complementing the shared generative process. In probabilistic modelling framework, adaptation may be expressed as conditioning the model on additional input examples serving as induction bias. Notable steps in this direction have been made by Rezende et al. (2016) whose model was able to condition on a single object to produce new examples of the concept it represents. Later, Edwards & Storkey (2016) proposed a model that maintained a global latent variable capturing statistics about multiple input objects which was used to condition the generative distribution. It allowed to implement the
fast learning ability, but due to the particular model architecture used the model was not well-suited to datasets consisting of several different concepts.
In this work we present Generative Matching Networks, a new family of conditional generative models capable of instant adaptation to new concepts that were not available at the training time but share the structure of underlying generative process with the training examples. By conditioning on additional inputs, Generative Matching Networks improve their predictive performance, the quality of generated samples and also adapt their latent space which may be useful for unsupervised feature extraction. Importantly, no explicit limitations on the conditioning data are imposed such as number of objects or number of different concepts which expands the applicability of one-shot generative modelling and distinguish our work from existing approaches. Our model is inspired by the attentional mechanism implemented in Matching Networks (Vinyals et al., 2016) previously proposed for discriminative tasks and the recent advances from meta-learning (Santoro et al., 2016). Our approach for adaptation is an extension of these ideas to generative modelling and it may be re-used in a variety of different models being not restricted to the particular architecture used in the paper. The source code for generative matching networks is available at http://github.com/sbos/gmn.
This paper is organized as follows. First, in section 2 we revisit the necessary background in variational approach to training generative models and mention the related work in conditional generative models. Then, in section 3 we describe the proposed generative model, it’s recognition counterpart and the training protocol. Section 4 contains experimental evaluation of the proposed model as both generative model and unsupervised feature extractor in small-shot learning settings. We conclude with discussion of the results in section 5.
2 BACKGROUND
We consider the problem of learning a probabilistic generative model which can be expressed as a probability distribution p(x|θ) over objects of interests x parametrized by θ. The major class of generative models introduce also latent variables z that are used to explain or generate an object x such that p(x|θ) = ∫ p(z|θ)p(x|z,θ)dz and assumed to be non-observable.
Currently, the common practice is to restrict the conditional distributions p(z|θ) and p(x|z,θ) to tractable distribution families and use deep neural networks for regressing their parameters. The expressive power of deep non-linear generative models comes at a price since neither marginal distribution p(x|θ) can be computed analytically nor it can be directly optimized in a statistically efficient way. Fortunately, intractable maximum likelihood training can be avoided in practice by resorting to adversarial training (Gutmann & Hyvärinen, 2012; Goodfellow et al., 2014) or variational inference framework (Kingma & Welling, 2013; Rezende et al., 2014) which we consider further.
2.1 TRAINING GENERATIVE MODELS WITH VARIATIONAL INFERENCE
Recent developments in variational inference alleviate problems with maximizing the intractable marginal likelihood log p(x|θ) by approximating it with a lower bound (Jordan et al., 1999):
log p(x|θ) ≥ L(θ,φ) = Eq [log p(x, z|θ)− log q(z|x,φ)] = log p(x|θ)− KL(q||p(·|x,θ)). (1)
Tightness of the bound is controlled by the recognition model q(z|x,φ) which aims to minimize Kullback-Leibler divergence from the true posterior p(z|x,θ). Similarly to the generative model, recognition model may also be implemented with the use of deep neural networks or other parameter regression which is known as amortized inference (Gershman & Goodman, 2014). Amortized inference allows to use a single recognition model for many training examples. Thus, it is convenient to perform training of the generative model p(x|θ) by stochastic gradient optimization of variational lower bounds (1) corresponding to independent observations {xi}Ni=1:
N∑ i=1 log p(xi|θ) ≥ N∑ i=1 Eq [log p(xi, zi|θ)− log q(zi|xi,φ)]→ max θ,φ .
The clear advantage of this approach is its scalability. Every stochastic update to the parameters computed from only a small portion of training examples has an immediate effect for the whole
dataset. However, while a single parameter update may be relatively fast a large number of them is required to significantly improve generative or inferential performance of the model.
Hence, gradient training of generative models usually results into an extensive computational process which prevents from rapid incremental learning. In the next section we discuss potential solutions to this problem that allow to implement fast learning ability in generative models.
2.2 ADAPTATION IN GENERATIVE MODELS
In probabilistic modelling framework the natural way of incorporating knowledge about newly available data is conditioning. One may design a model that being conditioned on the additional input data X = x1,x2, . . . ,xT represents a new generative distribution p(x|X,θ). An implementation of this idea can be found in the model by Rezende et al. (2016). Besides many other attractive novelties such as using sophisticated attention and feedback components, the model was able to produce new examples of a concept that was missing at the training time but had similarities in the underlying generative process with the other training examples. The model supported an explicit conditioning on a single observation x′ representing the new concept to construct a new generative distribution of the form p(x|x′,θ). The explicit conditioning when adaptation is performed by the model itself and and has to be learned is not the only way to propagate knowledge about new data. Another solution which is often encountered in Bayesian models is to maintain a global latent variable α encoding information about the whole available dataset such that the individual observations are conditionally independent given it’s value. The model then would have the following form:
p(X|θ) = ∫ p(α|θ) ∏T t=1 p(xt|α,θ)dα. (2)
The principal existence of such a global variable may be justified by the de Finetti’s theorem (Diaconis & Freedman, 1980) under the exchangeability assumption. In the model (2), the conditional generative distribution p(x|X,θ) is then defined implicitly via posterior over the global variable:
p(x|X,θ) = ∫ p(x|α,θ)p(α|X,θ)dα. (3)
Once there is an efficient inference procedure for the global variable α, fast adaptation of the generative model can be implemented straightforwardly.
There are several relevant examples of generative models with global latent variables used for model adaptation and one-shot learning. Salakhutdinov et al. (2013) combined deep Boltzmann machine (DBM) with nested Dirichlet process (nDP) in a Hierarchical-Deep (HD) model. While being a compelling demonstration of important ideas from Bayesian nonparametrics and deep learning, the HD model required an extensive Markov chain Monte Carlo inference procedure used both for training and adaptation. Thus, while Bayesian learning approach could prevent overfitting the fast learning ability still presents a challenge for sampling-based inference.
Later, Lake et al. (2015) proposed Bayesian program learning (BPL) approach for building a generative model of handwritten characters. The model was defined as a probabilistic program contained fine-grained specification of prior knowledge of the task such as generation of strokes and their composition into characters mimicking human drawing strategies. Authors used an extensive posterior inference as the training procedure and the conditioning mechanism (3) for generating new examples. The model was shown to efficiently learn from small number of training examples, but similarly to the HD model, sophisticated and computationally expensive inference procedure makes fast adaptation in BPL generally hard to achieve.
The recently proposed neural statistician model (Edwards & Storkey, 2016) is an example of deep generative model with a global latent variable (2). The model was trained by optimizing a variational lower bound following the approach described in section 2.1 but with an additional recognition model approximating posterior distribution over the global latent variable. Authors designed the recognition model to be computationally efficient and require only a single pass over data which consisted of extracting special features from the examples, applying to them a pooling operation (e.g. averaging) and passing the result to another network providing parameters of the variational approximation. This simple architecture allowed for the fast learning and guaranteed invariance to both data permutations and size of the conditioning dataset. However, experimentally the fast
learning ability in the model was evaluated only in the setting where all of the training examples represented the same single concept.
We argue that in order to capture more information about the conditioning data such as a number of different concepts a more sophisticated aggregation procedure must be employed. Moreover, a fixed parametric description is too restrictive for an accurate representation of datasets of varying size. This motivates us to combine the best of two worlds: nonparametric representation of data and fast inference with neural recognition models. We proceed with a description of the proposed model.
3 GENERATIVE MATCHING NETWORKS
Generative matching networks aim to model conditional generative distributions of the form p(x|X,θ). Similarly to other deep generative models we introduce a local latent variable z. Thus the full joint distribution of our model can be expressed as:
p(x, z|X,θ) = p(z|X,θ)p(x|z,X,θ). (4)
We also maintain a recognition model approximating the posterior over the latent variable z: q(z|x,X,φ) ≈ p(z|x,X,θ). In order to design a fast adaptation mechanism we have to make certain assumptions about relationships between training data and the new data used to condition the model. Thus we assume the homogeneity of generative processes for training and conditioning data up to some parametrization. One may think of this parametrization as specifying weights of a neural network defining a generative model. The generative process is assumed to have an approximately linear dependence on the parameters such that interpolation between parameters corresponding to different examples of the same concept can serve as good parameters for generating other examples. A similar assumption is used e.g. in the neural statistician model (Edwards & Storkey, 2016).
However, even if a single concept can be well embedded to a fixed parameter space, this does not imply that a diverse set of concepts will fit into the same parametrization. Hence we express the dependency on the conditioning data in a different way. Instead of embedding the whole conditioning dataset we use a special matching procedure that extracts relevant observations from X and interpolates between their descriptions allowing to generate and recognize similar observations.
3.1 BASIC MODEL
In the basic model, the prior over latent variables p(z) is independent from conditioning data X, e.g. a standard normal distribution. In order to generate a new object, a sample from the prior z and conditioning objects X = x1,x2, . . . ,xT are mapped into the matching space Φ where they are compared using a similarity function sim(., .) to form an attention kernel a(z,x). After that, the conditioning objects are interpolated in the prototype space Ψ weighted according to the attention kernel. The resulting interpolation is then used to parametrize the generative process that corresponds to the sampled value of latent variable.
Formally, the described matching procedure can be described by the following equations:
r = T∑ t=1 a(z,xt)ψL(xt), a(z,xt) = exp(sim(fL(z), gL(xt)))∑T t′=1 exp(sim(fL(z), gL(xt′))) . (5)
After the vector r is computed, it is used as an input to a decoder, e.g. a deconvolutional network.
Functions fL and gL are used to map latent variables and conditioning objects, correspondingly, into the matching space Φ. Since Φ is supposed to be a feature space that is good for discriminating between objects, gL can be implemented as a feature extractor suitable for the domain of observations, a convolutional network in our case. We found it sufficient to implement the function fL as a simple affine transformation followed by a non-linearity, because the latent variable itself is assumed to be an abstract object description. We also used a simple dot product as a similarity function between these vectors.
Function ψL can also be considered as a feature extractor, although since the features useful to specify the generative process are not necessarily good for discrimination, it makes sense to represent
ψL and gL differently. However, in our implementation ψL was implemented as a convolutional network sharing most of the parameters with gL to keep the number of trainable parameters small.
We have described the basic matching procedure on the example of the conditional likelihood p(x|z,X,θ). Although the procedure (5) is invoked in several parts of the model, each part may operate with it’s own implementation of the functions, hence the subscript ·L used for the functions f , g and ψ is for likelihood part and below we use ·R to denote the recognition part. The recognition model q(z|X,x) uses the matching procedure (5) with the difference that the conditioning objects are being matched not with a value of latent variable, but rather with an observation x. The feature extractor fR in this case can share most of the parameters with gR and in our implementation these functions were identical for matching in the recognition model, i.e. gR = fR Moreover, since gL is also used to project observations into the space Φ, we further re-use already defined functionality by setting gR = gL. We also shared prototype functions ψ for all parts of our model although this is not technically required.
After the matching, interpolated prototype vector r is used to compute parameters of the approximate posterior which in our case was a normal distribution with diagonal covariance matrix, i.e. q(z|X,x,φ) = N (z|µ(r),Σ(r)). A major difference between the generative matching networks and the originally proposed discriminative matching networks (Vinyals et al., 2016) is that since no label information is available to the model, the interpolation in equation (5) is performed not in the label space but rather in the prototype space which itself is defined by the model and is learnt during the training.
One can note that the described conditional model is not applicable in a situation where no conditioning objects are available. A possible solution to this problem involves implicit addition of a pseudo-input to the set of conditioning objects X. A pseudo-input is not an actual observation, but rather just the corresponding outputs of functions f , g and ψ which are assumed to be another trainable parameters.
A stochastic computational graph describing the basic model with pseudo-input can be found on figure 1. Further by default we assume the presence of a single pseudo-input in the model and denote models without pseudo-input as conditional.
3.2 EXTENSIONS
Although the basic model is capable of instant adaptation to the conditioning dataset X, it admits a number of extensions that can seriously improve it’s performance.
The disadvantage of the basic matching procedure (5) is that conditioning observations X are embedded to the space Φ independently from each other. Similarly to discriminative matching networks we address this problem by computing full contextual embeddings (FCE) (Vinyals et al., 2015). In order to obtain a joint embedding of conditioning data we allow K attentional passes over X of the form (5), guided by a recurrent controller R which accumulates global knowledge about the conditioning data in its hidden state h. The hidden state is thus passed to feature extractors f and g to obtain context-dependent embeddings.
We refer to this process as the full matching procedure which modifies equation (5) as:
rk = T∑ t=1 a(z,xt)ψ(xt), a(z,xt) = exp(sim(f(z,hk), g(xt,hk)))∑T t′=1 exp(sim(f(z,hk), g(xt′ ,hk))) ,
hk+1 = R(hk, rk).
(6)
The output of the full matching procedure is thus the interpolated prototype vector from the last iteration rK and the last hidden state of hK+1.
Besides context-dependent embedding of the conditioning data, full matching procedure allows to implement the data-dependent prior over latent variables p(z|X). In this case, no query point such as a latent variable z or an observation x is used to match with the conditioning data and only hidden state of the controller h is passed to functions f and g. Output of the procedure is then used to compute parameters of the prior, i.e. means and standard deviations in our case.
As we discuss in the experiments section, we found these extensions so important that further we consider only the model with full matching described by equation (6) and data-dependent prior. Please refer to the appendix and the source code for architectural details of our implementation.
3.3 TRAINING
Training of our model consists of maximizing marginal likelihood of a dataset X which can be expressed as:
p(X|θ) = T∏
t=1
p(xt|X<t,θ), X<t = {xs}t−1s=1. (7)
Ideally we would like to use the whole available training data as X but due to computational limitations we instead use a training strategy rooted in curriculum learning (Bengio et al., 2009) and meta-learning (Thrun, 1998; Vilalta & Drissi, 2002; Hochreiter et al., 2001) which recently was successfully applied for one-shot discriminative learning (Santoro et al., 2016). In particular, we define a task-generating distribution pd(X) which in our case samples datasets X of size T from training examples. Then we train our model to explain well all of the sampled datasets simultaneously:
Epd(X) [p(X|θ)]→ maxθ . (8) Obviously, the structure of task-generating distribution has a large impact on training and using an arbitrary distribution will unlikely lead to good results. Hence, we assume that at the training time we have an access to label information and can distinguish different concepts or classes. We thus constrain pd(X) to generate datasets consisting of examples that represent up to C randomly selected classes so that even on short datasets the model has a clear incentive to re-use conditioning data. This may be considered as a form of weak supervision but we want to emphasize that one does not need the label information at test time unless the model is deliberately used for classification which is also possible.
Since the marginal likelihood (7) as well as the conditional marginal likelihoods are intractable we instead use variational lower bound (see section 2.1) as a proxy to p(X|θ) in the objective (8):
L(X,θ,φ) = ∑T
t=1 Eq(zt|xt,X<t,φ) [log p(xt, zt|X<t,θ)− log q(zt|xt,X<t,φ)] .
4 EXPERIMENTS
For our experiments we use the Omniglot dataset (Lake et al., 2015) which consists of 1623 classes of handwritten characters from 50 different alphabets. The first 30 alphabets are devoted for training and the remaining 20 alphabets are left for testing. Importantly, only 20 examples of each class are available which makes this dataset specifically useful for small-shot learning problems. Unfortunately, the literature is inconsistent in usage of the dataset and multiple versions of Omniglot were used for evaluation which differ by train/test split, resolution, binarization and augmentation, see e.g. (Burda et al., 2015; Rezende et al., 2016; Santoro et al., 2016).
We use the canonical split provided by Lake et al. (2015). In order to speed-up training we downscaled images to 28×28 resolution and since the result was fully binary we did not apply any further
pre-processing. We also did not augment our data in contrast to (Santoro et al., 2016; Edwards & Storkey, 2016) to make future comparisons with our results easier.
Unless otherwise stated, we train models on datasets of length T = 20 and of up to C = 2 different classes as we did not observe any improvement from training with larger values of C.
4.1 NUMBER OF ATTENTION STEPS
Since the full context matching procedure (6) described in section 3.2 consists of multiple attention steps, it is interesting to see the effect of these numbers on model’s performance. We trained several models with smaller architecture and T = 10 varying number of attention steps allowed for the likelihood and recognition shared controller and the prior controller respectively. The models were compared using exponential moving averages of lower bounds corresponding to different numbers of conditioning examples X<t obtained during the training. Results of the comparison can be found on figure 2.
Interestingly, larger numbers of steps lead to better results, however lower bounds are almost not improving after the shared controller is allowed for 4 steps. This behaviour was not observed with discriminative matching networks perhaps confirming the difficulty of unsupervised learning. Another important result is that the standard Gaussian prior makes adaptation significantly harder for the model yet still possible which justifies the importance of adaptation not just for the likelihood model but also for the prior.
One may also see that all models preferred to set higher variances for a prior resulting to higher entropy comparing to standard normal prior. Clearly as more examples are available, generative matching networks become more certain about the data and output less dispersed Gaussians.
Based on this comparison we decided to proceed with models that have 4 steps for the shared controller and a single step for the prior controller which is a reasonable compromise between computational cost and performance.
4.2 FAST ADAPTATION AND SMALL-SHOT GENERATION
In this section we compare generative matching networks with a set of baselines by expected conditional likelihoods Epd(X)p(xt|X<t). The conditional likelihoods were estimated using importance sampling with 1000 samples from the recognition model used as a proposal.
As we mention in section 3.1, it is possible to add a pseudo-input to the model to make it applicable for cases when no conditioning data is available. In this comparison by default we assume that a single pseudo-input was added to the model, otherwise we denote a model with no pseudo-input as conditional. When training and evaluating conditional models we ensure that the first C objects in a dataset belong to different classes so that they in principle contain enough information to explain rest of the dataset.
We found it hard to properly compute conditional likelihoods for the neural statistician model (3) and hence had to exclude this model from the comparison, please see appendix for the details. Instead, we consider a simple generative matching network denoted as avg in which the matching procedure is replaced with prototype averaging which makes the adaptation mechanism similar to the one used in neural statistician. We also omitted sequential generative models (Rezende et al., 2016) from the
comparison as they were reported to overfit on the canonical train/test split of Omniglot. Another baseline we use is a standard variational autoencoder which has the same architecture for generative and recognition model as the full generative matching networks.
Table 1 contains results of the evaluation on the test alphabets from Omniglot. Ctrain and Ctest denote the maximum number of classes in task-generating distributions pd(·) used for training and evaluating respectively.
As one could expect, larger values of Ctest make adaptation harder since on average less examples of the same class are available to the model. Still generative matching networks are capable of working in low-data regime even when testing setting is harder than one used for training, i.e. Ctest > Ctrain. Unsurprisingly, adaptation by averaging over prototype features performed reasonably well for simple datasets constructed of a single class, although significantly worse than the proposed matching procedure. On more difficult datasets with mixed examples of two different classes (Ctest = 2) averaging was ineffective for expressing dependency on the conditioning data which justifies our argument on the necessity of nonparametric representations.
In order to visually assess the fast adaptation ability of generative matching networks we also provide conditionally generated samples in figure 3. Interestingly, the conditional version of our model which does not use a pseudo-input both at training and testing time generated samples slightly more similar to the conditioning data while sacrificing the predictive performance. Therefore, presence or absence of the pseudo-input should depend on target application of the model, i.e. density estimation or producing new examples.
5 CONCLUSION
In this paper we presented a new class of conditional deep generative models called generative matching networks. These models are capable of fast adaptation to conditioning dataset by adjusting both the latent space and the predictive density while making very few assumptions on the data. The nonparametric matching enabling these features can be seen as a generalization of the original matching procedure since it allows a model to define the label space itself extending the applicability of matching networks to unsupervised and perhaps semi-supervised settings. We believe that these ideas can evolve further and help to implement more data-efficient models in other domains such as reinforcement learning where data acquisition is especially hard.
ACKNOWLEDGMENTS
We would like to thank Michael Figurnov and Timothy Lillicrap for useful discussions. Dmitry P. Vetrov is supported by RFBR project No.15-31-20596 (mol-a-ved) and by Microsoft: MSU joint research center (RPD 1053945).
APPENDIX A. MODEL ARCHITECTURE
CONDITIONAL GENERATOR
The conditional generator network producing parameters for p(x|z,X,θ) has concatenation of z and the output of the matching operation [r,h] as input which is transformed to 3 × 3 × 32 tensor and then passed through 3 residual blocks of transposed convolutions. Each block has the following form:
h = conv1(x), y = f(conv2(h) + h) + pool(scale(x)),
where f is a non-linearity which in our architecture is always parametric rectified linear function (He et al., 2015).
The block is parametrized by size of filters used in convolutions conv1 and conv2, shared number of filters F and stride S.
• scale is another convolution with 1× 1 filters and the shared stride S. • In all other convolutions number of filters is the same and equals F . • conv1 and pool have also stride S. • conv2 preserve size of the input by padding and has stride 1.
Blocks used in our paper have the following parameters (W1 ×H1,W2 ×H2, F, S):
1. (2× 2, 2× 2, 32, 2) 2. (3× 3, 3× 3, 16, 2) 3. (4× 4, 3× 3, 16, 2)
Then log-probabilities for binary pixels were obtained by summing the result of these convolutions along the channel dimension.
FEATURE ENCODER ψ
Function ψ has an architecture which is symmetric from the generator network. The only difference is that the scale scale operation is replaced by bilinear upscaling.
The residual blocks for feature encoder has following parameters:
1. (4× 4, 3× 3, 16, 2)
2. (3× 3, 3× 3, 16, 2)
3. (2× 2, 2× 2, 32, 2)
The result is a tensor of 3× 3× 32 = 288 dimensions.
FUNCTIONS f AND g
Each function f or g used in our model is simply an affine transformation of feature encoder’s output (interpreted as a vector) to a 200-dimensional space followed by parametric rectified non-linearity.
APPENDIX B. TRANSFER TO MNIST
In this experiment we test the ability of generative matching networks to adapt not just to new concepts, but also to a new domain. Since we trained our models on 28×28 resolution for Omniglot it should be possible to apply them on MNIST dataset as well. We used the test part of MNIST to which we applied a single random binarization.
Table 2 contains estimated predictive likelihood for different models. Qualitative results from the evaluation on Omniglot remain the same. Although transfer to a new domain caused significant drop in performance for all of the models, one may see that generative matching networks still demonstrate the ability to adapt to conditioning data. At the same time, average matching does not seem to efficiently re-use the conditioned data in such transfer task since relative improvements in expected conditional log-likelihood are rather small. Apparently, the model trained on a one-class datasets also learned highly dataset-dependent features as it actually performed even worse than the model with Ctrain = 2.
We also provide conditional samples on figure 4. Both visual quality of samples and test loglikelihoods are significantly worse comparing to Omniglot which can be caused by a visual difference of the MNIST digits from Omniglot characters. The images are bolder and less regular due to binarization. Edwards & Storkey (2016) suggest that the quality of transfer may be improved by augmentation of the training data, however for the sake of experimental simplicity and reproducibility we resisted from any augmentation.
APPENDIX C. CLASSIFICATION
Generative matching networks are useful not only as adaptive density estimators. For example, one may use a pre-trained model for classification in several ways. Given a small number of labeled examples Xc = {xc,1,xc,2, . . .xc,N} for each class c ∈ {1, 2, . . . , C}, it possible to use the probability p(x|Xc) as a relative score to assign class c for a new object x. Alternatively, one may use the recognition model q(z|X1, . . . ,XC) to extract features describing the new object x and then use a classifier of choice, e.g. the nearest neighbour classifier. We implemented this method using cosine similarity on mean parameters of approximate Normal posteriors.
The results under different number of training examples available are provided in table 3. Surprisingly, the simpler model with average matching performed slightly better than the full matching model. Perhaps, generative matching networks are very smooth density models and even being conditioned on a number of same-class example still assign enough probability mass to discrepant observations. The same conclusion can be made by assessing the generated samples on figure 3 which may guide further research on the topic.
APPENDIX D. EVALUATION OF THE NEURAL STATISTICIAN MODEL
The neural statistician model falls into the category of models with global latent variables which we describe in section 2.2. The conditional likelihood for these models has the form:
p(x|X,θ) = ∫ p(α|X,θ)p(x|α,θ)dα.
This quantity is hard to compute since it consists of an expectation with respect to the true posterior over global variable α. Since this distribution is intractable, simple importance sampling can not be used to estimate the likelihood. Thus, we tried the following strategies.
First, we used self-normalizing importance sampling to directly estimate p(x|X,θ) as p̂(x|X,θ) = ∑S s=1 wsp(x, z (s)|α(s),θ)∑S
s=1 ws , ws =
p(α(s),X,Z(s)|θ) q(α(s)|X,φ)q(Z(s), z(s)|X,x,α(s),φ) ,
but observed somewhat contradictory results such as non-monotonic dependency of the estimate on the size of conditioning dataset. The diagnostic of the effective sample size suggested that the recognition model is not well suited as proposal for the task.
Another strategy was to sequentially estimate p(X<t,θ) and then use the equation
p(xt|X<t,θ) = p(xt,X<t|θ) p(X<t|θ) ,
which appeared to as unreliable as the previous strategy.
This paper proposes an interesting idea for rapidly adapting generative models in the low data regime. The idea is to use similar techniques that are used in one-shot learning, specifically ideas from matching networks. To that end, the authors propose the generative matching networks model, which is effectively a variational auto-encoder that can be conditioned on an input dataset. Given a query point, the model matches the query point to points in the conditioning set using an attention model in an embedding space (this is similar to matching networks). The results on the Omniglot dataset show that this method is successfully able to rapidly adapt to new input distributions given few examples.
I think that the method is very interesting, however the major issue for me with this paper is a lack of clarity. I outline more details below, but overall I found the paper somewhat difficult to follow. There are a lot of details that I feel are scattered throughout, and I did not get a sense after reading this paper that I would be able to implement the method and replicate the results. My suggestion is to consolidate the major implementation details into a single section, and be explicit about the functional form of the different embedding functions and their variants.
I was a bit disappointed to see that weak supervision in the form of labels had to be used. How does the method perform in a completely unsupervised setting? This could be an interesting baseline.
There is a lack of definition of the different functions. Some basic insight into the functional forms of f, g, \phi, sim and R would be nice. Otherwise it is very unclear to me what’s going on.
Section 3.2: “only state of the recurrent controller was used for matching”, my reading of this section (after several passes) is that the pseudo-input is used in the place of a regular input. Is this correct? Otherwise, this sentence/section needs more clarification. I noticed upon further reading in section 4.2 that there are two versions of the model: one in which a pseudo input is used, and one in which a pseudo input is not used (the conditional version). What is the difference in functional form between these? That is, how do the formulas for the embeddings f and g change between these settings?
“since the result was fully contrastive we did not apply any further binarization” what does it mean for a result to be fully contrastive?
For clarity, the figures and table refer to the number of shots, but this is never defined. I assume this is T here. This should be made consistent.
Figure 2: why is the value of T only 9 in this case? What does it mean for it to be 0? It is stated earlier that T should go up to 20 (I assume #shot corresponds to T). It also looks like the results continue to improve with an increased number of steps, I would like to see the results for 5 and maybe 6 steps as well. Presumably there will come a point where you get diminishing returns.
Table 1: is the VAE a fair baseline? You mention that Ctest affects Pd() in the evaluation. The fact that the VAE does not have an associated Ctest implies that the two models are being evaluated with a different metric. Can the authors clarify this? It’s important that the comparison is apples-to-apples.
MNIST is much more common than Omniglot for evaluating generative models. Would it be possible to perform similar experiments on this dataset? That way it can be compared with many more models.
Further, why are the negative log-likelihood values monotonically decreasing in the number of shots? That is, is there ever a case where increasing the number of shots can hurt things? What happens at T=30? 40?
As a minor grammatical issue, the paper is missing determiners in several sentences. At one point, the model is referred to as “she” instead of “it”. “On figure 3” should be changed to “in figure 3” in the experiments section.