GENERATIVE MATCHING NETWORKS
Despite recent advances, the remaining bottlenecks in deep generative models are necessity of extensive training and difficulties with generalization from small number of training examples. Both problems may be addressed by conditional generative models that are trained to adapt the generative distribution to additional input data. So far this idea was explored only under certain limitations such as restricting the input data to be a single object or multiple objects representing the same concept. In this work we develop a new class of deep generative model called generative matching networks which is inspired by the recently proposed matching networks for one-shot learning in discriminative tasks and the ideas from meta-learning. By conditioning on the additional input dataset, generative matching networks may instantly learn new concepts that were not available during the training but conform to a similar generative process, without explicit limitations on the number of additional input objects or the number of concepts they represent. Our experiments on the Omniglot dataset demonstrate that generative matching networks can significantly improve predictive performance on the fly as more additional data is available to the model and also adapt the latent space which is beneficial in the context of feature extraction.
1 INTRODUCTION
Deep generative models are currently one of the most promising directions in generative modelling. In this class of models the generative process is defined by a composition of conditional distributions modelled using deep neural networks which form a hierarchy of latent and observed variables. This approach allows to build models with complex, non-linear dependencies between variables and efficiently learn the variability across training examples.
Such models are trained by stochastic gradient methods which can handle large datasets and a wide variety of model architectures but also present certain limitations. The training process usually consists of small, incremental updates of networks’ parameters and requires many passes over training data. Notably, once a model is trained it cannot be adapted to newly available data without complete re-training to avoid catastrophic interference (McCloskey & Cohen, 1989; Ratcliff, 1990). There is also a risk of overfitting for concepts that are not represented by enough training examples which is caused by high capacity of the models. Hence, most of deep generative models are not well-suited for rapid learning in one-shot scenario which is often encountered in real-world applications where data acquisition is expensive or fast adaptation to new data is required.
A potential solution to these problems is explicit learning of adaptation mechanisms complementing the shared generative process. In probabilistic modelling framework, adaptation may be expressed as conditioning the model on additional input examples serving as induction bias. Notable steps in this direction have been made by Rezende et al. (2016) whose model was able to condition on a single object to produce new examples of the concept it represents. Later, Edwards & Storkey (2016) proposed a model that maintained a global latent variable capturing statistics about multiple input objects which was used to condition the generative distribution. It allowed to implement the
fast learning ability, but due to the particular model architecture used the model was not well-suited to datasets consisting of several different concepts.
In this work we present Generative Matching Networks, a new family of conditional generative models capable of instant adaptation to new concepts that were not available at the training time but share the structure of underlying generative process with the training examples. By conditioning on additional inputs, Generative Matching Networks improve their predictive performance, the quality of generated samples and also adapt their latent space which may be useful for unsupervised feature extraction. Importantly, no explicit limitations on the conditioning data are imposed such as number of objects or number of different concepts which expands the applicability of one-shot generative modelling and distinguish our work from existing approaches. Our model is inspired by the attentional mechanism implemented in Matching Networks (Vinyals et al., 2016) previously proposed for discriminative tasks and the recent advances from meta-learning (Santoro et al., 2016). Our approach for adaptation is an extension of these ideas to generative modelling and it may be re-used in a variety of different models being not restricted to the particular architecture used in the paper. The source code for generative matching networks is available at http://github.com/sbos/gmn.
This paper is organized as follows. First, in section 2 we revisit the necessary background in variational approach to training generative models and mention the related work in conditional generative models. Then, in section 3 we describe the proposed generative model, it’s recognition counterpart and the training protocol. Section 4 contains experimental evaluation of the proposed model as both generative model and unsupervised feature extractor in small-shot learning settings. We conclude with discussion of the results in section 5.
2 BACKGROUND
We consider the problem of learning a probabilistic generative model which can be expressed as a probability distribution p(x|θ) over objects of interests x parametrized by θ. The major class of generative models introduce also latent variables z that are used to explain or generate an object x such that p(x|θ) = ∫ p(z|θ)p(x|z,θ)dz and assumed to be non-observable.
Currently, the common practice is to restrict the conditional distributions p(z|θ) and p(x|z,θ) to tractable distribution families and use deep neural networks for regressing their parameters. The expressive power of deep non-linear generative models comes at a price since neither marginal distribution p(x|θ) can be computed analytically nor it can be directly optimized in a statistically efficient way. Fortunately, intractable maximum likelihood training can be avoided in practice by resorting to adversarial training (Gutmann & Hyvärinen, 2012; Goodfellow et al., 2014) or variational inference framework (Kingma & Welling, 2013; Rezende et al., 2014) which we consider further.
2.1 TRAINING GENERATIVE MODELS WITH VARIATIONAL INFERENCE
Recent developments in variational inference alleviate problems with maximizing the intractable marginal likelihood log p(x|θ) by approximating it with a lower bound (Jordan et al., 1999):
log p(x|θ) ≥ L(θ,φ) = Eq [log p(x, z|θ)− log q(z|x,φ)] = log p(x|θ)− KL(q||p(·|x,θ)). (1)
Tightness of the bound is controlled by the recognition model q(z|x,φ) which aims to minimize Kullback-Leibler divergence from the true posterior p(z|x,θ). Similarly to the generative model, recognition model may also be implemented with the use of deep neural networks or other parameter regression which is known as amortized inference (Gershman & Goodman, 2014). Amortized inference allows to use a single recognition model for many training examples. Thus, it is convenient to perform training of the generative model p(x|θ) by stochastic gradient optimization of variational lower bounds (1) corresponding to independent observations {xi}Ni=1:
N∑ i=1 log p(xi|θ) ≥ N∑ i=1 Eq [log p(xi, zi|θ)− log q(zi|xi,φ)]→ max θ,φ .
The clear advantage of this approach is its scalability. Every stochastic update to the parameters computed from only a small portion of training examples has an immediate effect for the whole
dataset. However, while a single parameter update may be relatively fast a large number of them is required to significantly improve generative or inferential performance of the model.
Hence, gradient training of generative models usually results into an extensive computational process which prevents from rapid incremental learning. In the next section we discuss potential solutions to this problem that allow to implement fast learning ability in generative models.
2.2 ADAPTATION IN GENERATIVE MODELS
In probabilistic modelling framework the natural way of incorporating knowledge about newly available data is conditioning. One may design a model that being conditioned on the additional input data X = x1,x2, . . . ,xT represents a new generative distribution p(x|X,θ). An implementation of this idea can be found in the model by Rezende et al. (2016). Besides many other attractive novelties such as using sophisticated attention and feedback components, the model was able to produce new examples of a concept that was missing at the training time but had similarities in the underlying generative process with the other training examples. The model supported an explicit conditioning on a single observation x′ representing the new concept to construct a new generative distribution of the form p(x|x′,θ). The explicit conditioning when adaptation is performed by the model itself and and has to be learned is not the only way to propagate knowledge about new data. Another solution which is often encountered in Bayesian models is to maintain a global latent variable α encoding information about the whole available dataset such that the individual observations are conditionally independent given it’s value. The model then would have the following form:
p(X|θ) = ∫ p(α|θ) ∏T t=1 p(xt|α,θ)dα. (2)
The principal existence of such a global variable may be justified by the de Finetti’s theorem (Diaconis & Freedman, 1980) under the exchangeability assumption. In the model (2), the conditional generative distribution p(x|X,θ) is then defined implicitly via posterior over the global variable:
p(x|X,θ) = ∫ p(x|α,θ)p(α|X,θ)dα. (3)
Once there is an efficient inference procedure for the global variable α, fast adaptation of the generative model can be implemented straightforwardly.
There are several relevant examples of generative models with global latent variables used for model adaptation and one-shot learning. Salakhutdinov et al. (2013) combined deep Boltzmann machine (DBM) with nested Dirichlet process (nDP) in a Hierarchical-Deep (HD) model. While being a compelling demonstration of important ideas from Bayesian nonparametrics and deep learning, the HD model required an extensive Markov chain Monte Carlo inference procedure used both for training and adaptation. Thus, while Bayesian learning approach could prevent overfitting the fast learning ability still presents a challenge for sampling-based inference.
Later, Lake et al. (2015) proposed Bayesian program learning (BPL) approach for building a generative model of handwritten characters. The model was defined as a probabilistic program contained fine-grained specification of prior knowledge of the task such as generation of strokes and their composition into characters mimicking human drawing strategies. Authors used an extensive posterior inference as the training procedure and the conditioning mechanism (3) for generating new examples. The model was shown to efficiently learn from small number of training examples, but similarly to the HD model, sophisticated and computationally expensive inference procedure makes fast adaptation in BPL generally hard to achieve.
The recently proposed neural statistician model (Edwards & Storkey, 2016) is an example of deep generative model with a global latent variable (2). The model was trained by optimizing a variational lower bound following the approach described in section 2.1 but with an additional recognition model approximating posterior distribution over the global latent variable. Authors designed the recognition model to be computationally efficient and require only a single pass over data which consisted of extracting special features from the examples, applying to them a pooling operation (e.g. averaging) and passing the result to another network providing parameters of the variational approximation. This simple architecture allowed for the fast learning and guaranteed invariance to both data permutations and size of the conditioning dataset. However, experimentally the fast
learning ability in the model was evaluated only in the setting where all of the training examples represented the same single concept.
We argue that in order to capture more information about the conditioning data such as a number of different concepts a more sophisticated aggregation procedure must be employed. Moreover, a fixed parametric description is too restrictive for an accurate representation of datasets of varying size. This motivates us to combine the best of two worlds: nonparametric representation of data and fast inference with neural recognition models. We proceed with a description of the proposed model.
3 GENERATIVE MATCHING NETWORKS
Generative matching networks aim to model conditional generative distributions of the form p(x|X,θ). Similarly to other deep generative models we introduce a local latent variable z. Thus the full joint distribution of our model can be expressed as:
p(x, z|X,θ) = p(z|X,θ)p(x|z,X,θ). (4)
We also maintain a recognition model approximating the posterior over the latent variable z: q(z|x,X,φ) ≈ p(z|x,X,θ). In order to design a fast adaptation mechanism we have to make certain assumptions about relationships between training data and the new data used to condition the model. Thus we assume the homogeneity of generative processes for training and conditioning data up to some parametrization. One may think of this parametrization as specifying weights of a neural network defining a generative model. The generative process is assumed to have an approximately linear dependence on the parameters such that interpolation between parameters corresponding to different examples of the same concept can serve as good parameters for generating other examples. A similar assumption is used e.g. in the neural statistician model (Edwards & Storkey, 2016).
However, even if a single concept can be well embedded to a fixed parameter space, this does not imply that a diverse set of concepts will fit into the same parametrization. Hence we express the dependency on the conditioning data in a different way. Instead of embedding the whole conditioning dataset we use a special matching procedure that extracts relevant observations from X and interpolates between their descriptions allowing to generate and recognize similar observations.
3.1 BASIC MODEL
In the basic model, the prior over latent variables p(z) is independent from conditioning data X, e.g. a standard normal distribution. In order to generate a new object, a sample from the prior z and conditioning objects X = x1,x2, . . . ,xT are mapped into the matching space Φ where they are compared using a similarity function sim(., .) to form an attention kernel a(z,x). After that, the conditioning objects are interpolated in the prototype space Ψ weighted according to the attention kernel. The resulting interpolation is then used to parametrize the generative process that corresponds to the sampled value of latent variable.
Formally, the described matching procedure can be described by the following equations:
r = T∑ t=1 a(z,xt)ψL(xt), a(z,xt) = exp(sim(fL(z), gL(xt)))∑T t′=1 exp(sim(fL(z), gL(xt′))) . (5)
After the vector r is computed, it is used as an input to a decoder, e.g. a deconvolutional network.
Functions fL and gL are used to map latent variables and conditioning objects, correspondingly, into the matching space Φ. Since Φ is supposed to be a feature space that is good for discriminating between objects, gL can be implemented as a feature extractor suitable for the domain of observations, a convolutional network in our case. We found it sufficient to implement the function fL as a simple affine transformation followed by a non-linearity, because the latent variable itself is assumed to be an abstract object description. We also used a simple dot product as a similarity function between these vectors.
Function ψL can also be considered as a feature extractor, although since the features useful to specify the generative process are not necessarily good for discrimination, it makes sense to represent
ψL and gL differently. However, in our implementation ψL was implemented as a convolutional network sharing most of the parameters with gL to keep the number of trainable parameters small.
We have described the basic matching procedure on the example of the conditional likelihood p(x|z,X,θ). Although the procedure (5) is invoked in several parts of the model, each part may operate with it’s own implementation of the functions, hence the subscript ·L used for the functions f , g and ψ is for likelihood part and below we use ·R to denote the recognition part. The recognition model q(z|X,x) uses the matching procedure (5) with the difference that the conditioning objects are being matched not with a value of latent variable, but rather with an observation x. The feature extractor fR in this case can share most of the parameters with gR and in our implementation these functions were identical for matching in the recognition model, i.e. gR = fR Moreover, since gL is also used to project observations into the space Φ, we further re-use already defined functionality by setting gR = gL. We also shared prototype functions ψ for all parts of our model although this is not technically required.
After the matching, interpolated prototype vector r is used to compute parameters of the approximate posterior which in our case was a normal distribution with diagonal covariance matrix, i.e. q(z|X,x,φ) = N (z|µ(r),Σ(r)). A major difference between the generative matching networks and the originally proposed discriminative matching networks (Vinyals et al., 2016) is that since no label information is available to the model, the interpolation in equation (5) is performed not in the label space but rather in the prototype space which itself is defined by the model and is learnt during the training.
One can note that the described conditional model is not applicable in a situation where no conditioning objects are available. A possible solution to this problem involves implicit addition of a pseudo-input to the set of conditioning objects X. A pseudo-input is not an actual observation, but rather just the corresponding outputs of functions f , g and ψ which are assumed to be another trainable parameters.
A stochastic computational graph describing the basic model with pseudo-input can be found on figure 1. Further by default we assume the presence of a single pseudo-input in the model and denote models without pseudo-input as conditional.
3.2 EXTENSIONS
Although the basic model is capable of instant adaptation to the conditioning dataset X, it admits a number of extensions that can seriously improve it’s performance.
The disadvantage of the basic matching procedure (5) is that conditioning observations X are embedded to the space Φ independently from each other. Similarly to discriminative matching networks we address this problem by computing full contextual embeddings (FCE) (Vinyals et al., 2015). In order to obtain a joint embedding of conditioning data we allow K attentional passes over X of the form (5), guided by a recurrent controller R which accumulates global knowledge about the conditioning data in its hidden state h. The hidden state is thus passed to feature extractors f and g to obtain context-dependent embeddings.
We refer to this process as the full matching procedure which modifies equation (5) as:
rk = T∑ t=1 a(z,xt)ψ(xt), a(z,xt) = exp(sim(f(z,hk), g(xt,hk)))∑T t′=1 exp(sim(f(z,hk), g(xt′ ,hk))) ,
hk+1 = R(hk, rk).
(6)
The output of the full matching procedure is thus the interpolated prototype vector from the last iteration rK and the last hidden state of hK+1.
Besides context-dependent embedding of the conditioning data, full matching procedure allows to implement the data-dependent prior over latent variables p(z|X). In this case, no query point such as a latent variable z or an observation x is used to match with the conditioning data and only hidden state of the controller h is passed to functions f and g. Output of the procedure is then used to compute parameters of the prior, i.e. means and standard deviations in our case.
As we discuss in the experiments section, we found these extensions so important that further we consider only the model with full matching described by equation (6) and data-dependent prior. Please refer to the appendix and the source code for architectural details of our implementation.
3.3 TRAINING
Training of our model consists of maximizing marginal likelihood of a dataset X which can be expressed as:
p(X|θ) = T∏
t=1
p(xt|X<t,θ), X<t = {xs}t−1s=1. (7)
Ideally we would like to use the whole available training data as X but due to computational limitations we instead use a training strategy rooted in curriculum learning (Bengio et al., 2009) and meta-learning (Thrun, 1998; Vilalta & Drissi, 2002; Hochreiter et al., 2001) which recently was successfully applied for one-shot discriminative learning (Santoro et al., 2016). In particular, we define a task-generating distribution pd(X) which in our case samples datasets X of size T from training examples. Then we train our model to explain well all of the sampled datasets simultaneously:
Epd(X) [p(X|θ)]→ maxθ . (8) Obviously, the structure of task-generating distribution has a large impact on training and using an arbitrary distribution will unlikely lead to good results. Hence, we assume that at the training time we have an access to label information and can distinguish different concepts or classes. We thus constrain pd(X) to generate datasets consisting of examples that represent up to C randomly selected classes so that even on short datasets the model has a clear incentive to re-use conditioning data. This may be considered as a form of weak supervision but we want to emphasize that one does not need the label information at test time unless the model is deliberately used for classification which is also possible.
Since the marginal likelihood (7) as well as the conditional marginal likelihoods are intractable we instead use variational lower bound (see section 2.1) as a proxy to p(X|θ) in the objective (8):
L(X,θ,φ) = ∑T
t=1 Eq(zt|xt,X<t,φ) [log p(xt, zt|X<t,θ)− log q(zt|xt,X<t,φ)] .
4 EXPERIMENTS
For our experiments we use the Omniglot dataset (Lake et al., 2015) which consists of 1623 classes of handwritten characters from 50 different alphabets. The first 30 alphabets are devoted for training and the remaining 20 alphabets are left for testing. Importantly, only 20 examples of each class are available which makes this dataset specifically useful for small-shot learning problems. Unfortunately, the literature is inconsistent in usage of the dataset and multiple versions of Omniglot were used for evaluation which differ by train/test split, resolution, binarization and augmentation, see e.g. (Burda et al., 2015; Rezende et al., 2016; Santoro et al., 2016).
We use the canonical split provided by Lake et al. (2015). In order to speed-up training we downscaled images to 28×28 resolution and since the result was fully binary we did not apply any further
pre-processing. We also did not augment our data in contrast to (Santoro et al., 2016; Edwards & Storkey, 2016) to make future comparisons with our results easier.
Unless otherwise stated, we train models on datasets of length T = 20 and of up to C = 2 different classes as we did not observe any improvement from training with larger values of C.
4.1 NUMBER OF ATTENTION STEPS
Since the full context matching procedure (6) described in section 3.2 consists of multiple attention steps, it is interesting to see the effect of these numbers on model’s performance. We trained several models with smaller architecture and T = 10 varying number of attention steps allowed for the likelihood and recognition shared controller and the prior controller respectively. The models were compared using exponential moving averages of lower bounds corresponding to different numbers of conditioning examples X<t obtained during the training. Results of the comparison can be found on figure 2.
Interestingly, larger numbers of steps lead to better results, however lower bounds are almost not improving after the shared controller is allowed for 4 steps. This behaviour was not observed with discriminative matching networks perhaps confirming the difficulty of unsupervised learning. Another important result is that the standard Gaussian prior makes adaptation significantly harder for the model yet still possible which justifies the importance of adaptation not just for the likelihood model but also for the prior.
One may also see that all models preferred to set higher variances for a prior resulting to higher entropy comparing to standard normal prior. Clearly as more examples are available, generative matching networks become more certain about the data and output less dispersed Gaussians.
Based on this comparison we decided to proceed with models that have 4 steps for the shared controller and a single step for the prior controller which is a reasonable compromise between computational cost and performance.
4.2 FAST ADAPTATION AND SMALL-SHOT GENERATION
In this section we compare generative matching networks with a set of baselines by expected conditional likelihoods Epd(X)p(xt|X<t). The conditional likelihoods were estimated using importance sampling with 1000 samples from the recognition model used as a proposal.
As we mention in section 3.1, it is possible to add a pseudo-input to the model to make it applicable for cases when no conditioning data is available. In this comparison by default we assume that a single pseudo-input was added to the model, otherwise we denote a model with no pseudo-input as conditional. When training and evaluating conditional models we ensure that the first C objects in a dataset belong to different classes so that they in principle contain enough information to explain rest of the dataset.
We found it hard to properly compute conditional likelihoods for the neural statistician model (3) and hence had to exclude this model from the comparison, please see appendix for the details. Instead, we consider a simple generative matching network denoted as avg in which the matching procedure is replaced with prototype averaging which makes the adaptation mechanism similar to the one used in neural statistician. We also omitted sequential generative models (Rezende et al., 2016) from the
comparison as they were reported to overfit on the canonical train/test split of Omniglot. Another baseline we use is a standard variational autoencoder which has the same architecture for generative and recognition model as the full generative matching networks.
Table 1 contains results of the evaluation on the test alphabets from Omniglot. Ctrain and Ctest denote the maximum number of classes in task-generating distributions pd(·) used for training and evaluating respectively.
As one could expect, larger values of Ctest make adaptation harder since on average less examples of the same class are available to the model. Still generative matching networks are capable of working in low-data regime even when testing setting is harder than one used for training, i.e. Ctest > Ctrain. Unsurprisingly, adaptation by averaging over prototype features performed reasonably well for simple datasets constructed of a single class, although significantly worse than the proposed matching procedure. On more difficult datasets with mixed examples of two different classes (Ctest = 2) averaging was ineffective for expressing dependency on the conditioning data which justifies our argument on the necessity of nonparametric representations.
In order to visually assess the fast adaptation ability of generative matching networks we also provide conditionally generated samples in figure 3. Interestingly, the conditional version of our model which does not use a pseudo-input both at training and testing time generated samples slightly more similar to the conditioning data while sacrificing the predictive performance. Therefore, presence or absence of the pseudo-input should depend on target application of the model, i.e. density estimation or producing new examples.
5 CONCLUSION
In this paper we presented a new class of conditional deep generative models called generative matching networks. These models are capable of fast adaptation to conditioning dataset by adjusting both the latent space and the predictive density while making very few assumptions on the data. The nonparametric matching enabling these features can be seen as a generalization of the original matching procedure since it allows a model to define the label space itself extending the applicability of matching networks to unsupervised and perhaps semi-supervised settings. We believe that these ideas can evolve further and help to implement more data-efficient models in other domains such as reinforcement learning where data acquisition is especially hard.
ACKNOWLEDGMENTS
We would like to thank Michael Figurnov and Timothy Lillicrap for useful discussions. Dmitry P. Vetrov is supported by RFBR project No.15-31-20596 (mol-a-ved) and by Microsoft: MSU joint research center (RPD 1053945).
APPENDIX A. MODEL ARCHITECTURE
CONDITIONAL GENERATOR
The conditional generator network producing parameters for p(x|z,X,θ) has concatenation of z and the output of the matching operation [r,h] as input which is transformed to 3 × 3 × 32 tensor and then passed through 3 residual blocks of transposed convolutions. Each block has the following form:
h = conv1(x), y = f(conv2(h) + h) + pool(scale(x)),
where f is a non-linearity which in our architecture is always parametric rectified linear function (He et al., 2015).
The block is parametrized by size of filters used in convolutions conv1 and conv2, shared number of filters F and stride S.
• scale is another convolution with 1× 1 filters and the shared stride S. • In all other convolutions number of filters is the same and equals F . • conv1 and pool have also stride S. • conv2 preserve size of the input by padding and has stride 1.
Blocks used in our paper have the following parameters (W1 ×H1,W2 ×H2, F, S):
1. (2× 2, 2× 2, 32, 2) 2. (3× 3, 3× 3, 16, 2) 3. (4× 4, 3× 3, 16, 2)
Then log-probabilities for binary pixels were obtained by summing the result of these convolutions along the channel dimension.
FEATURE ENCODER ψ
Function ψ has an architecture which is symmetric from the generator network. The only difference is that the scale scale operation is replaced by bilinear upscaling.
The residual blocks for feature encoder has following parameters:
1. (4× 4, 3× 3, 16, 2)
2. (3× 3, 3× 3, 16, 2)
3. (2× 2, 2× 2, 32, 2)
The result is a tensor of 3× 3× 32 = 288 dimensions.
FUNCTIONS f AND g
Each function f or g used in our model is simply an affine transformation of feature encoder’s output (interpreted as a vector) to a 200-dimensional space followed by parametric rectified non-linearity.
APPENDIX B. TRANSFER TO MNIST
In this experiment we test the ability of generative matching networks to adapt not just to new concepts, but also to a new domain. Since we trained our models on 28×28 resolution for Omniglot it should be possible to apply them on MNIST dataset as well. We used the test part of MNIST to which we applied a single random binarization.
Table 2 contains estimated predictive likelihood for different models. Qualitative results from the evaluation on Omniglot remain the same. Although transfer to a new domain caused significant drop in performance for all of the models, one may see that generative matching networks still demonstrate the ability to adapt to conditioning data. At the same time, average matching does not seem to efficiently re-use the conditioned data in such transfer task since relative improvements in expected conditional log-likelihood are rather small. Apparently, the model trained on a one-class datasets also learned highly dataset-dependent features as it actually performed even worse than the model with Ctrain = 2.
We also provide conditional samples on figure 4. Both visual quality of samples and test loglikelihoods are significantly worse comparing to Omniglot which can be caused by a visual difference of the MNIST digits from Omniglot characters. The images are bolder and less regular due to binarization. Edwards & Storkey (2016) suggest that the quality of transfer may be improved by augmentation of the training data, however for the sake of experimental simplicity and reproducibility we resisted from any augmentation.
APPENDIX C. CLASSIFICATION
Generative matching networks are useful not only as adaptive density estimators. For example, one may use a pre-trained model for classification in several ways. Given a small number of labeled examples Xc = {xc,1,xc,2, . . .xc,N} for each class c ∈ {1, 2, . . . , C}, it possible to use the probability p(x|Xc) as a relative score to assign class c for a new object x. Alternatively, one may use the recognition model q(z|X1, . . . ,XC) to extract features describing the new object x and then use a classifier of choice, e.g. the nearest neighbour classifier. We implemented this method using cosine similarity on mean parameters of approximate Normal posteriors.
The results under different number of training examples available are provided in table 3. Surprisingly, the simpler model with average matching performed slightly better than the full matching model. Perhaps, generative matching networks are very smooth density models and even being conditioned on a number of same-class example still assign enough probability mass to discrepant observations. The same conclusion can be made by assessing the generated samples on figure 3 which may guide further research on the topic.
APPENDIX D. EVALUATION OF THE NEURAL STATISTICIAN MODEL
The neural statistician model falls into the category of models with global latent variables which we describe in section 2.2. The conditional likelihood for these models has the form:
p(x|X,θ) = ∫ p(α|X,θ)p(x|α,θ)dα.
This quantity is hard to compute since it consists of an expectation with respect to the true posterior over global variable α. Since this distribution is intractable, simple importance sampling can not be used to estimate the likelihood. Thus, we tried the following strategies.
First, we used self-normalizing importance sampling to directly estimate p(x|X,θ) as p̂(x|X,θ) = ∑S s=1 wsp(x, z (s)|α(s),θ)∑S
s=1 ws , ws =
p(α(s),X,Z(s)|θ) q(α(s)|X,φ)q(Z(s), z(s)|X,x,α(s),φ) ,
but observed somewhat contradictory results such as non-monotonic dependency of the estimate on the size of conditioning dataset. The diagnostic of the effective sample size suggested that the recognition model is not well suited as proposal for the task.
Another strategy was to sequentially estimate p(X<t,θ) and then use the equation
p(xt|X<t,θ) = p(xt,X<t|θ) p(X<t|θ) ,
which appeared to as unreliable as the previous strategy.

This paper proposes an interesting idea for rapidly adapting generative models in the low data regime. The idea is to use similar techniques that are used in one-shot learning, specifically ideas from matching networks. To that end, the authors propose the generative matching networks model, which is effectively a variational auto-encoder that can be conditioned on an input dataset. Given a query point, the model matches the query point to points in the conditioning set using an attention model in an embedding space (this is similar to matching networks). The results on the Omniglot dataset show that this method is successfully able to rapidly adapt to new input distributions given few examples.
I think that the method is very interesting, however the major issue for me with this paper is a lack of clarity. I outline more details below, but overall I found the paper somewhat difficult to follow. There are a lot of details that I feel are scattered throughout, and I did not get a sense after reading this paper that I would be able to implement the method and replicate the results. My suggestion is to consolidate the major implementation details into a single section, and be explicit about the functional form of the different embedding functions and their variants.
I was a bit disappointed to see that weak supervision in the form of labels had to be used. How does the method perform in a completely unsupervised setting? This could be an interesting baseline.
There is a lack of definition of the different functions. Some basic insight into the functional forms of f, g, \phi, sim and R would be nice. Otherwise it is very unclear to me what’s going on.
Section 3.2: “only state of the recurrent controller was used for matching”, my reading of this section (after several passes) is that the pseudo-input is used in the place of a regular input. Is this correct? Otherwise, this sentence/section needs more clarification. I noticed upon further reading in section 4.2 that there are two versions of the model: one in which a pseudo input is used, and one in which a pseudo input is not used (the conditional version). What is the difference in functional form between these? That is, how do the formulas for the embeddings f and g change between these settings?
“since the result was fully contrastive we did not apply any further binarization” what does it mean for a result to be fully contrastive?
For clarity, the figures and table refer to the number of shots, but this is never defined. I assume this is T here. This should be made consistent.
Figure 2: why is the value of T only 9 in this case? What does it mean for it to be 0? It is stated earlier that T should go up to 20 (I assume #shot corresponds to T). It also looks like the results continue to improve with an increased number of steps, I would like to see the results for 5 and maybe 6 steps as well. Presumably there will come a point where you get diminishing returns.
Table 1: is the VAE a fair baseline? You mention that Ctest affects Pd() in the evaluation. The fact that the VAE does not have an associated Ctest implies that the two models are being evaluated with a different metric. Can the authors clarify this? It’s important that the comparison is apples-to-apples.
MNIST is much more common than Omniglot for evaluating generative models. Would it be possible to perform similar experiments on this dataset? That way it can be compared with many more models.
Further, why are the negative log-likelihood values monotonically decreasing in the number of shots? That is, is there ever a case where increasing the number of shots can hurt things? What happens at T=30? 40?
As a minor grammatical issue, the paper is missing determiners in several sentences. At one point, the model is referred to as “she” instead of “it”. “On figure 3” should be changed to “in figure 3” in the experiments section.


Recurrent neural networks have shown their ability to construct sentence or paragraph representations. Variants such as LSTM overcome the problem of vanishing gradients to some degree, thus being able to model long-time dependency. Still, these recurrent based models lack the ability of capturing complex semantic compositions. To address this problem, we propose a model which can incorporate local contexts with the guide of global context attention. Both the local and global contexts are obtained through LSTM networks. The working procedure of this model is just like how we human beings read a text and then answer a related question. Empirical studies show that the proposed model can achieve state of the art on some benchmark datasets. Attention visualization also verifies our intuition. Meanwhile, this model does not need pretrained embeddings to get good results.
1 INTRODUCTION
Sentiment classification, a key problem in sentiment analysis, has drawn lots of interest since 2000s (Pang & Lee, 2008). Traditional machine learning based methods often use one-hot features to represent a text. In these methods, each word (or n-gram) is treated as an independent token, and a text is represented as the words within. Though convenient intuitively, these methods cannot model the semantic dependencies between words, and often lead to the curse of dimensionality problem.
Neural models have shown their great performance in learning representation. Bengio et al. (2003) proposed the feed forward neural network language model. Ever since, many neural network based models have been widely used in many natural language processing (NLP) tasks, such as named entity recognization (Collobert & Weston, 2008), machine translation (Cho et al., 2014) and sentiment analysis (dos Santos & Gatti, 2014). Compared to one-hot representation, words are represented as distributed, dense vectors (also called word embeddings) in these neural models. Mikolov et al. (2013b) showed that, through proper training, these vectors can model syntactic and semantic relationships between words. More importantly, these vectors can be calculated naturally, which leads to another problem — semantic composition.
For many NLP tasks, we often face the problem of representing a sentence or paragragh using word embeddings of its containing words. Simply methods such as concatenation or weighted sum cannot give us satisfactory results. Many neural models, such as recursive neural network (Socher et al., 2011), recurrent neural network (Mikolov et al., 2010) and convolutional neural networks (Collobert & Weston, 2008) have been proposed to solve this problem.
Since a sentence or paragraph is naturally a sequence of word tokens, recurrent neural networks (RNN) can be used to composite word embeddings into a sentence or paragraph embedding. Recurrent networks utilize special units which are connected to themselves to maintain a hidden state. The internal hidden state can be seen as a representation of the contexts. However, basic recurrent network suffers from the vanishing or exploding gradients problem (Hochreiter et al., 2001), due to its direct connection between states in adjacent timesteps. Long short term memory (LSTM) network alleviates this problem by introducing a cell that has the ability to read, write or reset its internal state (Graves, 2012). Even so, LSTM has the bias of prefering recent inputs, making it difficult to capture the important information within a long sequence.
Recently, many studies have been done to help RNNs improve their semantic composition performance. A common way is to introduce another kind of network, such as CNN into RNN (Lai et al., 2015). In these combined models, different parts share complementary advantages. Lately, the attention mechanism (Bahdanau et al., 2014) has drawn a lot of attention. In an attention model, we utilize states in all timesteps, rather than simply force the network to represent the context into one fixed-length vector. More importantly, by visualizing the attention weights, we can get a deeper insight of how the network works (Bahdanau et al., 2014; Rocktäschel et al., 2015).
Inspired by the attention thought, we propose a framework which is similar to how we human beings read a text and then answer an related question. This framework consists of two parts: the first part is a bidirectional LSTM (Bi-LSTM) to learn a rough representation of the whole text, the second part is another Bi-LSTM with attention, which can learn local contexts and incorporate them to get a refined context representation.
2 RELATED WORK
In this section, we introduce the related work concerning sentiment analysis, neural semantic composition and attention mechanism.
2.1 SENTIMENT ANALYSIS
Sentiment analysis, also called opinion mining or subjectivity analysis, is the field of study that analyzes people’s sentiments or emotions towards various entities (Liu, 2012). Due to its great importance, sentiment analysis has a wide range of applications, such as finding opinions of certain products, predicting the stock market or political elections.
Generally, existing methods for sentiment classification fall into two categories: machine-learningbased methods and semantic-based methods. Semantic methods relies more on lexcial and linguistic resources. Pang et al. (2002) first tried using machine learning methods to classifiy texts based on their sentiments. Ever since, a lot more classification models and features have been tested. For machine learning procedures, feature design is always the key step for a good result. While in most cases, a well-designed feature needs much human effort and domain knowledge, which severely limits its applications.
Deep learning models have drawn lots of attention due to their automatic representation learning. Deep learning is about learning multiple levels of representations, obtained by combining non-linear modules (LeCun et al., 2015). Various models have been tested for sentiment analysis task, such as recursive autoencoder (Socher et al., 2011), paragraph vector (Le & Mikolov, 2014), convolutional network (Kim, 2014), etc.
2.2 NEURAL LANGUAGE MODELS AND SEMANTIC COMPOSITION
Bengio et al. (2003) proposed a feed-forward neural network language model, which used distributed representations for words. With the rapid progress of neural language models for NLP tasks, the distributed representations of words have become more and more important.
In most neural language models, the words (or even characters) are represented as dense, distributed vectors. These embeddings are usually used as the inputs of a network. The subsequent layers then conduct semantic composition operations over these word/character embeddings to obtain higher level semantic represetations. This kind of representations based on embeddings of different seman-
tic levels (character, word, phrase, sentence, paragraph, etc.) have shown their great performance on many NLP tasks.
After Bengio’s pioneering work, Collobert & Weston (2008) replaced the probabilistic output with a scoring scheme, and provided a unified architecture for many NLP tasks. Mnih & Hinton (2007) used a log-bilinear model to get the hidden representations. They future provided a hierarchical version (Mnih & Hinton, 2009), which uses a hierarchical softmax at the output layer to reduce calculation amount. Huang et al. (2012) added the global context information into C&W model, which can improve word embeddings as well as solve the polysemy problem. Their idea of using global context inspired our work, while in our work we use the global context as attention for the following training. Socher et al. (2011; 2012; 2013) proposed a family of recursive autoencoder models, which construct higher level represetations using a tree structure. Mikolov et al. (2013a) provided their word2vec (CBOW and Skip-gram) model. The word2vec model is actually a simple but effective log-linear model, imbibing advantages of previous neural models. Word embeddings pretrained using word2vec on a large corpus can be used for many other NLP tasks, often used as initializations or auxiliary features. Pennington et al. (2014) performed the training on aggregated global word-word co-occurrence statistics from a corpus, and their resulting global vectors (GloVe) showcase interesting linear substructures of the word vector space. Le & Mikolov (2014) extended the idea of word2vec model from word to paragragh, and the paragraph vectors they got can improve the performance on various NLP tasks.
Although seemingly different, these models all conduct semantic compositions directly or indirectly at intermediate layers. Currently, various neural models have been tried for semantic composition, such as multi-layer perceptron, recusive network, convolutional network and recurrent network. When representing higher level language units, convolutinal netwoks are better at capturing local information, while recurrent networks excel at storing history information. The network structure has also been explored, such as sequencial or tree structure (Li et al., 2015).
Of all these models, recurrent networks are highly valued due to its human alike reading style. More importantly, recurrent networks are capable of modeling sequencial inputs or outputs, which is essential for tasks such as machine translation (Cho et al., 2014). When applying recurrent networks for NLP tasks, it is typical to encode the semantic information of all words within a sentence into one fixed-length vector. The resulting representation is then used for generating related output words or simply used for classification.
2.3 ATTENTION MECHANISM
Attention mechanism is first used for machine translation (Bahdanau et al., 2014) in a encoderdecoder framework. The key idea of attention is to allow the network revisit all parts of a source sentence for an output decision, instead of trying to encode all information of a source sentence into a fixed-length vector. By this mechanism, we can get a deeper insight of how the network works. In the work of Bahdanau et al. (2014), they found the alignment between words by visualizing the attention weights.
Successive studies on attention mechanism include image caption (Xu et al., 2015), semantic entailment (Rocktäschel et al., 2015), aspect level sentiment analysis (Tang et al., 2016), etc. The most related work to ours is an attention model for sequence classification (Shen & Lee, 2016), in which they used LSTM output to weight word embeddings. While in our work, we use the global context to selectively choose the most important local contexts, which provides mutual promotions for both global and local context representations.
3 LEARNING TO UNDERSTAND: THE MODEL
It’s widely accepted that RNN alone, even with LSTM cells, cannot store long-time information. But, we may ask, do we really have to get a refined document or text embedding? Taking the reading comprehension task as an example, after a quick look at the text given, what we human beings got is also a rough memory of this text. Even so, this did not affect us to make a right decision. What we typically do next is to look at the problem, and rescan the text with a rough memory of the whole text in mind. With this global context information, we can easily locate the important parts for our
problem. In the end, even if we have not understood the whole text thoroughly, we can make right decisions for the problem in hand.
This thought inspired us to propose this attention model, which incorporates local contexts with a rough global context attention.
3.1 BASIC MODEL: BI-LSTM
Long short term memory network can be seen as a variant of the basic recurrent network, mainly to address the long-time dependency probem. It has special units that can pass their states to the next timestep. LSTM network replaces the hidden units with more flexible cells. The cell itself is a small network, which can decide its own reading, writing or resetting behavior.
The LSTM network we use is similar to Graves (2012). We don’t consider peephole connections. Given an input sequence s = (x1, x2, · · · , xn), (xi ∈ Rm), the update procedure from timestep t− 1 to t can be described as follows:
it = σ(W ixxt + U ihht−1 + b i) (1)
ft = σ(W fxxt + U fhht−1 + b f ) (2)
ot = σ(W oxxt + U ohht−1 + b o) (3)
C̃t = φ(W cxxt + U chht−1 + b c) (4)
Ct = ft Ct−1 + it C̃t (5) ht = ot φ(ct) (6)
where denotes the element-wise multiplication, it, ft, ot ∈ Rh stands for the input gate, forget gate, output gate respectively. The LSTM cell maintains an internal cell state. In one timestep, the input gate determines how much the outside information can come in, the output gate determines how much the internal information can go out, and the forget gate determines how much the internal information should be forgotten when passed to next timestep.
For a given sequence s = (x1, x2, · · · , xn), if we feed the token from left to right one by one into the LSTM network, the final internal state −→h will contain more information of the latter tokens due to the model bias. We feed the sequences backwards into the LSTM again, thus getting a backwardpropagating internal state←−h . The final representation of the global context is calculated by
h = −→ h ⊕←−h (7)
where ⊕ denotes concatenation. This is a typical bidirectional LSTM (Bi-LSTM) network. Again, we point out that this gloal context representation may be just a rough one. We use it as attention rather than directly treat it as a refined representation for the task afterwards.
3.2 TWO-SCAN APPROACH WITH ATTENTION
In this part, we introduce our Two-Scan Approach with Attention (TS-ATT) model.
The TS-ATT model contains two parts: a Bi-LSTM network first scans the whole text to get a global context represetation, another Bi-LSTM with attention scans the text again to get local contexts around each token, and composite them using the global context attention. The model are shown in Figure (1).
First Scan: we first use a Bi-LSTM network to gain a global context representation. As described in Sec. (3.1), the final global context vector h is a concatention of the final output of left-LSTM←−h and right-LSTM −→h . This global context serves as attention for the second scan. To distinguish the final context representation in the second scan, we denote h as hscan1, which is
hscan1 = ←− h ⊕−→h (8)
Second Scan: During the second scan of the text, we use another Bi-LSTM networks to learn local context for each word. Since already with the rough global context in mind, when looking at one word, we actually see its context meaning rather than the word itself. Thus, we use a word’s local context vector instead of its word embedding here. Inspired by the work of Lai et al. (2015), for word wi, we concatenate its left context c (l) i , right context c (r) i and embedding ei to construct its final local context represetation ci, which is:
c (l) i = f
( W (lc)c
(l) i+1 +W (le)ei
) (9)
c (r) i = f
( W (rc)c
(r) i−1 +W (re)ei
) (10)
ci = c (l) i ⊕ ei ⊕ c (r) i (11)
Attention: As we may expect, during the second scan, we don’t have to encode every word into the text representation. Just paying attention to the parts which are important for our problem in hand would be enough. Thus, we use the global context vector hscan1 as the attention of all the local contexts. Attention weights are acquired by one-layer feed-forward network, which is:
yatti = f ( W att(hscan1 ⊕ ci) + batt ) (12)
where W att and batt are attention parameters. All the attention weights are then fed into a softmax layer to obtain probabilistic attention weights. The final representation used for classification (denoted as hscan2) is the weighted sum of all the local context vectors, which is:
αi = exp(y (a) i )∑n
i=1 exp(y (a) i )
(13)
hscan2 =
n∑
i=1
αici (14)
The context vector hscan2 is then fed into a fully-connected layer and a softmax layer to do the classification task. The whole network is training to minimize the cross-entropy error E = 1N ∑N
i=1H(y|p) = − 1N ∑N i=1 ∑n j=1 yj log pj over all training examples, where p denotes the predictions and y the true labels. Parameters are updated using the RMSProp (Tieleman & Hinton, 2012) rule.
3.3 SINGLE-SCAN APPROACH WITH ATTENTION
In the proposed TS-ATT model, we use two Bi-LSTM netwoks to conduct these two scans. However, sometimes we may adopt another faster, but may equally good approach: a single scan. In this way, we first look at the problems to be solved, then scan the original text looking for clues. During this process, we maintain a rough gloal context and pay attention to the important parts simultaneously.
Inspired by this thought, we proposed the Single-Scan Approach with Attention (SS-ATT) model, which is a reduced version of TS-ATT model. In this model, the final outputs of the second BiLSTM are directly used as attention, saving the need of the first Bi-LSTM network. The SS-ATT model is shown in Figure (2).
4 EXPERIMENTS
We evaluated our models on some benchmark datasets for sentiment classification. These datasets includes:
• Amazon1 by Blitzer et al. (2007). Amazon dataset contains product reviews from Amazon.com from many product types. This dataset is origially used for domain adaption, here, we only use it for open-domain classification, regardless of the domain difference.
• IMDB2 by Maas et al. (2011), a large movie review dataset collected from imdb.com website.
• Yelp 20133 User reviews and recommendations dataset, which are built by Tang et al. (2015).
The statistical information of these datasets are listed in Table (1).
For IMDB, the original dataset already has a train/test split, we stick to this behavior in our experiments. For Amazon, we split 20% of the data as the test set. For Yelp datasets, they also have a train/validation/test split.
For datasets without an explicit validation split (like IMDB and Amazon), we randomly select 10% of the training data as the validation data, which is used for hyperparameter tuning and early-stop in training.
4.1 BASELINES
We compare our attention model with some models related to our work. These models contain traditional machine learning methods such as support vector machine (SVM) as well as neural models such as LSTM.
Traditional machine learning methods:
(1) SVM+BOW: Support Vector Machine with Bag Of Words representation. SVM uses hinge loss to maximize the margin between different classes.
(2) NB+BOW: Naive Bayes with Bag Of Words representation. NB chooses the class with the largest posterior probability, which is calculated through priori probability and likelihood based on training data.
Neural model methods:
(3) CNN: Convolutional Neural Network for sentence classification model proposed by Kim (2014). We use 100 filters to evaluation this model.
(4) LSTM: Long Short Term Memory network (Graves, 2012)
(5) Bi-LSTM: Bi-directional LSTM network (Graves, 2012)
(6) RCNN: Recurrent Convolutional Neural Network proposed by Lai et al. (2015), which uses a bi-directional recurrent network to replace the convolution operation in a typical convolutional network.
(7) NAM: Neural Attention Model proposed by Shen & Lee (2016). This model uses LSTM output as attention directly for word embeddings.
4.2 SETUP
For texts of these datasets, we split them into word tokens using punctuations and spaces as delimiters. Unlike some studies earlier, we process each text as a single unit rather than split them into sentences, which actually provides us more challenges. We process texts this way mainly to test the model’s ability to construct representation of long texts, which requires the model to learn long-time dependency.
For the neural models, we set the word embedding dimention to be 300, and LSTM state dimention to be 300. For training, we use stochastic gradient descent based on mini-batches with a batch size of 32. The training process is monitored by the validation accuracy for early-stopping. After training, the model is evaluated by a separate test dataset.
Due to the different pre-processing and model parameters selecting scheme, our results may not look the same as they are reported in other papers. For a fair comparison, we rerun all the models using the same platform of our experiment. No tricks such as dropout, batch-normaliztion are used, since we want a comparison of these models themselves. Similarly, we use random initial word embeddings instead of pretrained embeddings such as word2vec (Mikolov et al., 2013a).
4.3 RESULTS AND DISCUSSIONS
Our experimental results are shown in Table (2).
As shown in Table (2), our methods (TS-ATT or SS-ATT) achieve the best accuracy on two of three datasets (IMDN and Yelp2013). For Amazon dataset, the result of our method is almost as good as the best model (CNN). These results demonstrate the effectiveness of our proposed models.
From Tabel (2), we can see CNN shows good performance in these sentiment classification tasks. But, CNN has many hyper-parameters to tune, such as filter numbers, filter size, pooling size, etc. Changes of these hyper-parameters affect CNN’s performance to a large scale. Zhang & Wallace (2015) did a sensitivity analysis of CNNs for sentence classification. Compared with CNNs, recurrent based networks have fewer hypermeters to fine-tune. Even so, basic neural network such as LSTM doesn’t perform so well, as shown in Table (2). Bi-LSTM fails to make significant improvements on these tasks.
Comparing with RCNN, which uses a max-pooling operation after an Bi-LSTM, our model surpass it on all three datasets (1.87%, 3.99% and 1.45% respectively). This result demonstrates the effectiveness of using global context (though a rough one) as attention. Further, our SS-ATT model can be seen as a single Bi-LSTM network with global attention. The great improvement compared to Bi-LSTM also witnesses the key role that global attention plays.
Comparing with NAM, which use global attention directly on word embeddings, our model also achieves big or small improvements (4.02%, 0.72% and 1.61% respectively). These improvements over NAM model indicate that incorporating local contexts with attention would be more suitable for text understanding.
As for our proposed models, we can see TS-ATT and SS-ATT can achieve approximately equally good results. The key difference of these two models is that SS-ATT uses only one Bi-LSTM, improving global and local context representation mutually when training. These results are consistent with our intuition of human-like reading style. Imagine when in a reading comprehension test, we may skip reading the whole texts at first, but directly look at the texts after knowing the questions. In this way, we are actually using single-scan approach to save time. But as long as we have the big picture (global context) in mind, this kind of process won’t harm our right decisions.
5 ATTENTION VISUALIZATION AND CASE STUDY
In order to get a deeper insight into our global-local context attention model, we track the attention weights when the model is trying to make a decision. Take the IMDB dataset as example, when evaluating our model, we accumulate the attention weights for each word appeared, then calculate their average attentions. By this way, we can get a better knowledge about what kind of words this model favors when trying to predict the sentiment of a text. The words with largest average attention are shown in Table (3). We select those words with a minimal appearing frequency (for IMDB we set it to 100, change of this number won’t affect the results so much).
From Table (3), we can see that our attention model can effectively find the most discriminative features. Word such as surprised is actually negative in most cases, while in our model it’s classified as positive correctly, since we usually use surprised to express our pleasance when some movie is beyond our expectation. Also, we notice that the word recommended is favored by
both polarities. In this case, a model has to rely its local context (such as not recommended or recommended) to make a right decision. This will be illustrated in following case study.
Since our method incorporates local contexts to work, we next provide some case studies where important local contexts are automatically selected for right decisions. We use cases where the word recommended serving a high attention and then dive into its surrounding words to support our thought above.
Figure (3) shows the attention weights of each word in the sample sentence segment or music of the and greeks - this movie is strongly recommended for lovers of the music. Figure (4) shows another negative sentence segment the new zealand film commission fourthly - a friend recommended it to me - however i was utterly disappointed.
-10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4
words around "recommended"
0.003
0.004
0.005
0.006
0.007
0.008
0.009
a tt
e n ti
o n w
e ig
h ts
ormusic
of the
and
greeks this movie is strongly recommended
for
lovers of
the
Figure 3: A positive example -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 words around "recommended"
0.003
0.004
0.005
0.006
0.007
0.008
0.009
0.010
a tt
e n ti
o n w
e ig
h ts
the new
zealand
film commission fourthly a friend recommended
it to
me
however
i
was
utterly
disappointed
the
Figure 4: A negtive example
From Figure (3) and Figure (4), we can see that the model actually concentrations on important local contexts rather than single embeddings. This further verifies our initial idea that, using a global context to incorporate local contexts is a suitable way for sentiment or more general text classification task.
6 CONCLUSIONS
In this paper, we proposed a global-local context attention framework for sentiment analysis. This method is similar to human’s reading behavior in a reading comprehension situation. First, this model uses a Bi-LSTM network to extract a global context representation. This global representation maybe inaccurate, but we only use it as attention for important local parts. In a second scan of the text, the model automatically finds the most important local contexts and incorporates them to make a final decision. A simple version of this model only needs one-scan of text, which can reduce the model complexity but almost equally effective. Experimental results demonstrate that our model performs better than existing related models. Using global context representation as attention, our model can effectively find the most important local parts to make a right decision.
ACKNOWLEDGMENTS
This research is supported by the Key Program of National Natural Science Foundation of China (Grant nos. U1536201 and U1405254), the National Natural Science Foundation of China (Grant no. 61472092), the National High Technology Research and Development Program of China (863 Program) (Grant no. 2015AA020101), the National Science and Technology Support Program of China (Grant no. 2014BAH41B00), and the Initiative Scientific Research Program of Tsinghua University.

This paper presents a hierarchical attention-based method for document classification.
The main idea is to first run a bidirectional LSTM to get global context vector, and then run another attention-based bidirectional LSTM that uses the final hidden state from the first pass to weight local context vectors (TS-ATT).
A simpler architecture that removes the first LSTM and uses the output of the second LSTM as the global context vector is also proposed (SS-ATT).
Experiments on three datasets are presented, however the results are mostly not state-of-the-art.
I think the idea is nice, but the experiment results are not convincing enough to justify this new model architecture.
Why is your Yelp 2013 dataset smaller than the original Tang et al, 2015 paper that has ~300k documents?
I noticed your other datasets are also quite small. Is it because your model is difficult to scale to large datasets?
You should also include results from Tang et al., 2015 in Table 2 that achieves 65.1% accuracy on Yelp 2013 (why is your number so much lower?)
I also suggest removing phrases such as "Learning to Understand" when presenting their model.
Overall, I think that this submission is a better fit for the workshop.
Minor comments:
- gloal -> global
- Not needing a pretrained embeddings, while of course nice, is not that big of a deal. Various models will work just fine without pretrained embeddings.