ENERGY-BASED SPHERICAL SPARSE CODING
In this paper, we explore an efficient variant of convolutional sparse coding with unit norm code vectors where reconstruction quality is evaluated using an inner product (cosine distance). To use these codes for discriminative classification, we describe a model we term Energy-Based Spherical Sparse Coding (EB-SSC) in which the hypothesized class label introduces a learned linear bias into the coding step. We evaluate and visualize performance of stacking this encoder to make a deep layered model for image classification.
1 INTRODUCTION
Sparse coding has been widely studied as a representation for images, audio and other vectorial data. This has been a highly successful method that has found its way into many applications, from signal compression and denoising (Donoho, 2006; Elad & Aharon, 2006) to image classification (Wright et al., 2009), to modeling neuronal receptive fields in visual cortex (Olshausen & Field, 1997). Since its introduction, subsequent works have brought sparse coding into the supervised learning setting by introducing classification loss terms to the original formulation to encourage features that are not only able to reconstruct the original signal but are also discriminative (Jiang et al., 2011; Yang et al., 2010; Zeiler et al., 2010; Ji et al., 2011; Zhou et al., 2012; Zhang et al., 2013).
While supervised sparse coding methods have been shown to find more discriminative features leading to improved classification performance over their unsupervised counterparts, they have received much less attention in recent years and have been eclipsed by simpler feed-forward architectures.
This is in part because sparse coding is computationally expensive. Convex formulations of sparse coding typically consist of a minimization problem over an objective that includes a least-squares (LSQ) reconstruction error term plus a sparsity inducing regularizer.
Because there is no closed-form solution to this formulation, various iterative optimization techniques are generally used to find a solution (Zeiler et al., 2010; Bristow et al., 2013; Yang et al., 2013; Heide et al., 2015). In applications where an approximate solution suffices, there is work that learns non-linear predictors to estimate sparse codes rather than solve the objective more directly (Gregor & LeCun, 2010). The computational overhead for iterative schemes becomes quite significant when training discriminative models due to the demand of processing many training examples necessary for good performance, and so sparse coding has fallen out of favor by not being able to keep up with simpler non-iterative coding methods.
In this paper we introduce an alternate formulation of sparse coding using unit length codes and a reconstruction loss based on the cosine similarity. Optimal sparse codes in this model can be computed in a non-iterative fashion and the coding objective lends itself naturally to embedding in a discriminative, energy-based classifier which we term energy-based spherical sparse coding (EBSSC). This bi-directional coding method incorporates both top-down and bottom-up information where the features representation depends on both a hypothesized class label and the input signal. Like Cao et al. (2015), our motivation for bi-directional coding comes from the “Biased Competition Theory”, which suggests that visual processing can be biased by other mental processes (e.g., topdown influence) to prioritize certain features that are most relevant to current task. Fig. 1 illustrates the flow of computation used by our SSC and EB-SSC building blocks compared to a standard feed-forward layer.
Our energy based approach for combining top-down and bottom-up information is closely tied to the ideas of Larochelle & Bengio (2008); Ji et al. (2011); Zhang et al. (2013); Li & Guo (2014)— although the model details are substantially different (e.g., Ji et al. (2011) and Zhang et al. (2013) use sigmoid non-linearities while Li & Guo (2014) use separate representations for top-down and bottom-up information). The energy function of Larochelle & Bengio (2008) is also similar but includes an extra classification term and is trained as a restricted Boltzmann machine.
1.1 NOTATION
Matrices are denoted as uppercase bold (e.g., A), vectors are lowercase bold (e.g., a), and scalars are lowercase (e.g., a). We denote the transpose operator with ᵀ, the element-wise multiplication operator with , the convolution operator with ∗, and the cross-correlation operator with ?. For vectors where we dropped the subscript k (e.g., d and z), we refer to a super vector with K components stacked together (e.g., z = [zᵀ1 , . . . , z ᵀ K ] ᵀ).
2 ENERGY-BASED SPHERICAL SPARSE CODING
Energy-based models capture dependencies between variables using an energy function that measure the compatibility of the configuration of variables (LeCun et al., 2006). To measure the compatibility between the top-down and bottom-up information, we define the energy function of EB-SSC to be the sum of bottom-up coding term and a top-down classification term:
E(x, y, z) = Ecode(x, z) + Eclass(y, z). (1) The bottom-up information (input signal x) and the top-down information (class label y) are tied together by a latent feature map z.
2.1 BOTTOM-UP RECONSTRUCTION
To measure the compatibility between the input signal x and the latent feature maps z, we introduce a novel variant of sparse coding that is amenable to efficient feed-forward optimization. While the idea behind this variant can be applied to either patch-based or convolutional sparse coding, we specifically use the convolutional variant that shares the burden of coding an image among nearby overlapping dictionary elements. Using such a shift-invariant approach avoids the need to learn dictionary elements which are simply translated copies of each other, freeing up resources to discover more diverse and specific filters (see Kavukcuoglu et al. (2010)).
Convolutional sparse coding (CSC) attempts to find a set of dictionary elements {d1, . . . ,dK} and corresponding sparse codes {z1, . . . , zK} so that the resulting reconstruction, r = ∑K k=1 dk ∗ zk accurately represents the input signal x. This is traditionally framed as a least-squares minimization with a sparsity inducing prior on z:
arg min z ‖x− K∑ k=1 dk ∗ zk‖22 + β‖z‖1. (2)
Unlike standard feed-forward CNN models that convolve the input signal x with the filters, this energy function corresponds to a generative model where the latent feature maps {z1, . . . , zK} are convolved with the filters and compared to the input signal (Bristow et al., 2013; Heide et al., 2015; Zeiler et al., 2010).
To motivate our novel variant of CSC, consider expanding the squared reconstruction error ‖x − r‖22 = ‖x‖22 − 2xᵀr + ‖r‖22. If we constrain the reconstruction r to have unit norm, the reconstruction error depends entirely on the inner product between x and r and is equivalent to the cosine similarity (up to additive and multiplicative constants). This suggests the closely related unit-length reconstruction problem:
arg max z xᵀ ( K∑ k=1 dk ∗ zk ) − β‖z‖1 (3)
s.t. ∥∥ K∑ k=1 dk ∗ zk ∥∥ 2 ≤ 1
In Appendix A we show that, given an optimal unit length reconstruction r̄∗ with corresponding codes z̄∗, the solution to the least squares reconstruction problem (Eq. 2) can be computed by a simple scaling r∗ = (xᵀr̄∗ − β2 ‖z̄
∗‖1)r̄∗. The unit-length reconstruction problem is no easier than the original least-squares optimization due to the constraint on the reconstruction which couples the codes for different filters. Instead consider a simplified constraint on z which we refer to as spherical sparse coding (SSC):
arg max ‖z‖2≤1 Ecode(x, z) = arg max ‖z‖2≤1 xᵀ ( K∑ k=1 dk ∗ zk ) − β‖z‖1. (4)
In 2.3 below, we show that the solution to this problem can be found very efficiently without requiring iterative optimization.
This problem is a relaxation of convolutional sparse coding since it ignores non-orthogonal interactions between the dictionary elements1. Alternately, assuming unit norm dictionary elements, the code norm constraint can be used to upper-bound the reconstruction length. We have by the triangle and Young’s inequality that:∥∥∑
k
dk ∗ zk ∥∥ 2 ≤ ∑ k ‖dk ∗ zk‖2 ≤ ∑ k ‖dk‖1‖zk‖1 ≤ D ∑ k ‖zk‖2 (5)
where the factor D is the dimension of zk and arises from switching from the 1-norm to the 2-norm. Since D ∑ k ‖zk‖2 ≤ 1 is a tighter constraint we have
max ‖ ∑ k dk∗zk‖2≤1 Ecode(x, z) ≥ max∑ k ‖zk‖2≤ 1 D Ecode(x, z) (6)
However, this relaxation is very loose, primarily due to the triangle inequality. Except in special cases (e.g., if the dictionary elements have disjoint spectra) the SSC codes will be quite different from the standard least-squares reconstruction.
1We note that our formulation is also closely related to the dynamical model suggested by Rozell et al. (2008), but without the dictionary-dependent lateral inhibition between feature maps. Lateral inhibition can solve the unit-length reconstruction formulation of standard sparse coding but requires iterative optimization.
2.2 TOP-DOWN CLASSIFICATION
To measure the compatibility between the class label y and the latent feature maps z, we use a set of one-vs-all linear classifiers. To provide more flexibility, we generalize this by splitting the code vector into positive and negative components:
zk = z + k + z − k z + k ≥ 0 z − k ≤ 0
and allow the linear classifier to operate on each component separately. We express the classifier score for a hypothesized class label y by:
Eclass(y, z) = K∑ k=1 w+ᵀy z + k + K∑ k=1 w−ᵀy z − k . (7)
The classifier thus is parameterized by a pair of weight vectors (w+yk and w − yk) for each class label y and k-th channel of the latent feature map.
This splitting, sometimes referred to as full-wave rectification, is useful since a dictionary element and its negative do not necessarily have opposite visual semantics. This splitting also allows the classifier the flexibility to assign distinct meanings or alternately be completely invariant to contrast reversal depending on the problem domain. For example, Shang et al. (2016) found CNN models with ReLU non-linearities which discard the negative activations tend to learn pairs of filters which are related by negation. Keeping both positive and negative responses allowed them to halve the number of dictionary elements.
We note that it is also straightforward to introduce spatial average pooling prior to classification by introducing a fixed linear operator P used to pool the codes (e.g., w+ᵀy Pz + k ). This is motivated by a variety of hand-engineered feature extractors and sparse coding models, such as Ren & Ramanan (2013), which use spatially pooled histograms of sparse codes for classification. This fixed pooling can be viewed as a form of regularization on the linear classifier which enforces shared weights over spatial blocks of the latent feature map. Splitting is also quite important to prevent information loss when performing additive pooling since positive and negative components of zk can cancel each other out.
2.3 CODING
Bottom-up reconstruction and top-down classification each provide half of the story, coupled by the latent feature maps. For a given input x and hypothesized class y, we would like to find the optimal activations z that maximize the joint energy function E(x, y, z). This requires solving the following optimization:
arg max ‖z‖2≤1 xᵀ ( K∑ k=1 dk ∗ zk ) − β‖z‖1 + K∑ k=1 w+ᵀyk z + k + K∑ k=1 w−ᵀyk z − k , (8)
where x ∈ RD is an image and y ∈ Y is a class hypothesis. zk ∈ RF is the k-th component latent variable being inferred; z+k and z − k are the positive and negative coefficients of zk, such that zk = z + k + z − k . The parameters dk ∈ RM , w + yk ∈ RF , and w − yk ∈ RF are the dictionary filter, positive coefficient classifier, and negative coefficient classifier for the k-th component respectively. A key aspect of our formulation is that the optimal codes can be found very efficiently in closedform—in a feed-forward manner (see Appendix B for a detailed argument).
2.3.1 ASYMMETRIC SHRINKAGE
To describe the coding processes, let us first define a generalized version of the shrinkage function commonly used in sparse coding. Our asymmetric shrinkage is parameterized by upper and lower thresholds −β− ≤ β+
shrink(β+,β−)(v) =  v − β + if v − β+ > 0
0 otherwise v + β− if v + β− < 0
(9)
Fig. 2 shows a visualization of this function which generalizes the standard shrinkage proximal operator by allowing for the positive and negative thresholds. In particular, it corresponds to the proximal operator for a version of the `1-norm that penalizes the positive and negative components with different weights |v|asym = β+‖v+‖1 +β−‖v−‖1. The standard shrink operator corresponds to shrink(β,−β)(v) while the rectified linear unit common in CNNs is given by a limiting case shrink(0,−∞)(v). We note that −β− ≤ β+ is required for shrink(β+,β−) to be a proper function (see Fig. 2).
2.3.2 FEED-FORWARD CODING
We now describe how codes can be computed in a simple feed-forward pass. Let
β+yk = β −w + yk, β − yk = β −w − yk (10)
be vectors of positive and negative biases whose entries are associated with a spatial location in the feature map k for class y. The optimal code z can be computed in three sequential steps:
1. Cross-correlate the data with the filterbank dk ? x
2. Apply an asymmetric version of the standard shrinkage operator
z̃k = shrink(β+yk,β − yk) (dk ? x) (11)
where, with abuse of notation, we allow the shrinkage function (Eq. 9) to apply entries in the vectors of threshold parameter pairs β+yk,β − yk to the corresponding elements of the argument.
3. Project onto the feasible set of unit length codes
z∗ = z̃
‖z̃‖2 . (12)
2.3.3 RELATIONSHIP TO CNNS:
We note that this formulation of coding has a close connection to single layer convolutional neural network (CNN). A typical CNN layer consists of convolution with a filterbank followed by a nonlinear activation such as a rectified linear unit (ReLU). ReLUs can be viewed as another way of inducing sparsity, but rather than coring the values around zero like the shrink function, ReLU truncates negative values. On the other hand, the asymmetric shrink function can be viewed as the sum of two ReLUs applied to appropriately biased inputs:
shrink(β+,β−)(x) = ReLU(x− β+)− ReLU(−(x+ β−)),
SSC coding can thus be seen as a CNN in which the ReLU activation has been replaced with shrinkage followed by a global normalization.
3 LEARNING
We formulate supervised learning using the softmax log-loss that maximizes the energy for the true class label yi while minimizing energy of incorrect labels ȳ.
arg min d,w+,w−,β≥0
α 2 (‖w+‖22 + ‖w−‖22 + ‖d‖22)
+ 1
N N∑ i=1 [− max ‖z‖2≤1 E(xi, yi, z) + log ∑ ȳ∈Y max ‖z̄‖2≤1 eE(xi,ȳ,z̄)]
s.t. − (β −w−yk) ≤ (β −w + yk) ∀y, k
, (13)
where α is the hyperparameter regularizing w+y , w − y , and d. We constrain the relationship between β and the entries of w+y and w − y in order for the asymmetric shrinkage to be a proper function (see Sec. 2.3.1 and Appendix B for details).
In classical sparse coding, it is typical to constrain the `2-norm of each dictionary filter to unit length. Our spherical coding objective behaves similarly. For any optimal code z∗, there is a 1-dimensional subspace of parameters for which z∗ is optimal given by scaling d inversely to w, β. For simplicity of the implementation, we opt to regularize d to assure a unique solution. However, as Tygert et al. (2015) point out, it may be advantageous from the perspective of optimization to explicitly constrain the norm of the filter bank.
Note that unlike classical sparse coding, where β is a hyperparameter that is usually set using crossvalidation, we treat it as a parameter of the model that is learned to maximize performance.
3.1 OPTIMIZATION
In order to solve Eq. 13, we explicitly formulate our model as a directed-acyclic-graph (DAG) neural network with shared weights, where the forward-pass computes the sparse code vectors and the backward-pass updates the parameter weights. We optimize the objective using stochastic gradient descent (SGD).
As mentioned in Sec. 2.3 shrinkage function is assymetric with parameters β+yk or β − yk as defined in Eq. 10. However, the inequality constraint on their relationship to keep the shrinkage function a proper function is difficult to enforce when optimizing with SGD. Instead, we introduce a central offset parameter and reduce the ordering constraint to pair of positivity constraints. Let
ŵ+yk = β + yk − bk ŵ − yk = β − yk + bk (14)
be the modified linear “classifiers” relative to the central offset bk. It is straightforward to see that if β+yk and β − yk that satisfy the constrain in Eq. 13, then adding the same value to both sides of the inequality will not change that. However, taking bk to be a midpoint between them, then both β+yk − bk and β − yk + bk will be strictly non-negative.
Using this variable substitution, we rewrite the energy function (Eq. 1) as
E′(x, y, z) = xᵀ ( K∑ k=1 dk ∗ zk ) + K∑ k=1 bk1 ᵀzk − K∑ k=1 ŵ+ᵀyk z + k + K∑ k=1 ŵ−ᵀyk z − k . (15)
where b is constant offset for each code channel. The modified linear “classification” terms now take on a dual role of inducing sparsity and measuring the compatibility between z and y.
This yields a modified learning objective that can easily be solved with existing implementations for learning convolutional neural nets:
arg min d,ŵ+,ŵ−,b
α 2 (‖ŵ+‖22 + ‖ŵ−‖22 + ‖d‖22)
+ 1
N N∑ i=1 [− max ‖z‖2≤1 E′(xi, yi, z) + log ∑ ȳ∈Y max ‖z̄‖2≤1 eE ′(xi,ȳ,z̄)]
s.t. ŵ+yk, ŵ − yk 0 ∀y, k
, (16)
where ŵ+ and ŵ− are the new sparsity inducing classifiers, and b are the arbitrary origin points. In particular, adding the K origin points allows us to enforce the constraint by simply projecting ŵ+ and ŵ− onto the positive orthant during SGD.
3.1.1 STACKING BLOCKS
We also examine stacking multiple blocks of our energy function in order to build a hierarchical representation. As mentioned in Sec. 3.1.1, the optimal codes can be computed in a simple feedforward pass—this applies to shallow versions of our model. When stacking multiple blocks of our energy-based model, solving for the optimal codes cannot be done in a feed-forward pass since the codes for different blocks are coupled (bilinearly) in the joint objective. Instead, we can proceed in an iterative manner, performing block-coordinate descent by repeatedly passing up and down the hierarchy updating the codes. In this section we investigate the trade-off between the number of passes used to find the optimal codes for the stacked model and classification performance.
For this purpose, we train multiple instances of a 2-block version of our energy-based model that differ in the number of iterations used when solving for the codes. For recurrent networks such as this, inference is commonly implemented by “unrolling” the network, where the parts of the network structure are repeated with parameters shared across these repeated parts to mimic an iterative algorithm that stops at a fixed number of iterations rather than at some convergence criteria.
In Fig. 3, we compare the performance between models that were unrolled zero to four times. We see that there is a difference in performance based on how many sweeps of the variables are made. In terms of the training objective, more unrolling produces models that have lower objective values with convergence after only a few passes. In terms of testing error, however, we see that full code inference is not necessarily better, as unrolling once or twice has lower errors than unrolling three or four times. The biggest difference was between not unrolling and unrolling once, where both the training objective and testing error goes down. The testing error decreases from 0.0131 to 0.0074. While there is a clear benefit in terms of performance for unrolling at least once, there is also a trade-off between performance and computational resource, especially for deeper models.
4 EXPERIMENTS
We evaluate the benefits of combining top-down and bottom-up information to produce classspecific features on the CIFAR-10 (Krizhevsky & Hinton, 2009) dataset using a deep version of our EB-SSC. All experiments were performed using MatConvNet (Vedaldi & Lenc, 2015) framework with the ADAM optimizer (Kingma & Ba, 2014). The data was preprocessed and augmented following the procedure in Goodfellow et al. (2013). Specifically, the data was made zero mean and whitened, augmented with horizontal flips (with a 0.5 probability) and random cropping. No weight decay was used, but we used a dropout rate of 0.3 before every convolution layer except for the first. For these experiments we consider a single forward pass (no unrolling).
4.1 CLASSIFICATION
We compare our proposed EB-SSC model to that of Springenberg et al. (2015), which uses rectified linear units (ReLU) as its non-linearity. This model can be viewed as a basic feed-forward version of our proposed model which we take as a baseline. We also consider variants of the baseline model that utilize a subset of architectural features of our proposed model (e.g., concatenated rectified linear units (CReLU) and spherical normalization (SN)) to understand how subtle design changes of the network architecture affects performance.
We describe the model architecture in terms of the feature extractor and classifier. Table 1 shows the overall network architecture of feature extractors, which consist of seven convolution blocks and two pooling layers. We test two possible classifiers: a simple linear classifier (LC) and our energy-based classifier (EBC), and use softmax-loss for all models. For linear classifiers, a numerical subscript indicates which of the seven conv blocks of the feature extractor is used for classification (e.g., LC7 indicates the activations out of the last conv block is fed into the linear classifier). For energy-based classifiers, a numerical subscript indicates which conv blocks of the feature extractor are replace with a energy-based classifier (e.g., EBC6−7 indicates the activations out of conv5 is fed into the energy-based classifier and the energy-based classifier has a similar architecture to the conv blocks it replaces). The notation differ because for energy-based classifiers, the optimal activations are a function of the hypothesized class label, whereas for linear classifiers, they are not.
The results shown in Table 2 compare our proposed model to the baselines ReLU+LC7 (Springenberg et al., 2015) and CReLU+LC7 (Shang et al., 2016), and to intermediate variants. The baseline models all perform very similarly with some small reductions in error rates over the baseline CReLU+LC7. However, CReLU+LC7 reduces the error rate over ReLU+LC7 by more than one percent (from 11.40% to 10.17%), which confirms the claims by Shang et al. (2016) and demonstrates the benefits of splitting positive and negative activations. Likewise, we see further decrease in the error rate (to 9.74%) from using spherical normalization. Though normalizing the activations doesn’t add any capacity to the model, this improved performance is likely because scale-invariant activations makes training easier. On the other hand, further sparsifying the activations yielded no
benefit. We tested values β = {0.001, 0.01} and found 0.001 to perform better. Replacing the linear classifier with our energy-based classifier further decreases the error rate by another half percent (to 9.23%).
4.2 DECODING CLASS-SPECIFIC CODES
A unique aspect of our model is that it is generative in the sense that each layer is explicitly trying to encode the activation pattern in the prior layer. Similar to the work on deconvolutional networks built on least-squares sparse coding (Zeiler et al., 2010), we can synthesize input images from activations in our spherical coding network by performing repeated deconvolutions (transposed convolutions) back through the network. Since our model is energy based, we can further examine how the topdown information of a hypothesized class effects the intermediate activations.
The first column in Fig. 4 visualizes reconstructions of a given input image based on activations from different layers of the model by convolution transpose. In this case we put in zeros for class biases (i.e., no top-down) and are able to recover high fidelity reconstructions of the input. In the remaining columns, we use the same deconvolution pass to construct input space representations of the learned classifier biases. At low levels of the feature hierarchy, these biases are spatially smooth since the receptive fields are small and there is little spatial invariance capture in the activations. At higher levels these class-conditional bias fields become more tightly localized.
Finally, in Fig. 5 we shows decodings from the conv2 and conv5 layer of the EB-SSC model for a given input under different class hypotheses. Here we subtract out the contribution of the top-down bias term in order to isolate the effect of the class conditioning on the encoding of input features. As visible in the figure, the modulation of the activations focused around particular regions of the image and the differences across class hypotheses becomes more pronounced at higher layers of the network.
5 CONCLUSION
We presented an energy-based sparse coding method that efficiently combines cosine similarity, convolutional sparse coding, and linear classification. Our model shows a clear mathematical connection between the activation functions used in CNNs to introduce sparsity and our cosine similarity convolutional sparse coding formulation. Our proposed model outperforms the baseline model and we show which attributes of our model contributes most to the increase in performance. We also demonstrate that our proposed model provides an interesting framework to probe the effects of class-specific coding.
APPENDIX A
Here we show that spherical sparse coding (SSC) with a norm constraint on the reconstruction is equivalent to standard convolutional sparse coding (CSC). Expanding the least squares reconstruction error and dropping the constant term ‖x‖2 gives the CSC problem:
max z 2xᵀ ( K∑ k=1 dk ∗ zk ) − ‖ K∑ k=1 dk ∗ zk‖22 − β K∑ k=1 ‖zk‖1.
Let = ‖ ∑K k=1 dk ∗ zk‖2 be the norm of the reconstruction for some code z and let u be the reconstruction scaled to have unit norm so that:
u =
∑K k=1 dk ∗ zk
‖ ∑K k=1 dk ∗ zk‖2 = K∑ k=1 dk ∗ z̄k with z̄ = 1 z
We rewrite the least-squares objective in terms of these new variables:
max z̄, >0 g(z̄, ) = max z̄, >0
2xᵀ ( u ) − ‖ u‖22 − β‖ z̄‖1
= max z̄, >0
2 ( xᵀu− β
2 ‖z̄‖1
) − 2
Taking the derivative of g w.r.t. yields the optimal scaling ∗ as a function of z̄:
(z̄)∗ = xᵀu− β 2 ‖z̄‖1.
Plugging (z̄)∗ back into g yields:
max z̄, >0 g(z̄, ) = max z̄,‖u‖2=1
( xᵀu− β
2 ‖z̄‖1
)2 .
Discarding solutions with < 0 can be achieved by simply dropping the square which results in the final constrained problem:
arg max z̄ xᵀ ( K∑ k=1 dk ∗ z̄k ) − β 2 K∑ k=1 ‖z̄k‖1
s.t. ‖ K∑ k=1 dk ∗ z̄k‖2 ≤ 1.
APPENDIX B
We show in this section that coding in the EB-SSC model can be solved efficiently by a combination of convolution, shrinkage and projection, steps which can be implemented with standard libraries on a GPU. For convenience, we first rewrite the objective in terms of cross-correlation rather than convolution (i.e., , xᵀ(dk ∗ zk) = (dk ? x)ᵀzk). For ease of understanding, we first consider the coding problem when there is no classification term.
z∗ = arg max ‖z‖22≤1 vᵀz− β‖z‖1,
where v = [(d1 ? x)ᵀ, . . . , (dK ? x)ᵀ]ᵀ. Pulling the constraint into the objective, we get its Lagrangian function: L(z, λ) = vᵀz− β‖z‖1 + λ ( 1− ‖z‖22 ) .
From the partial subderivative of the Lagrangian w.r.t. zi we derive the optimal solution as a function of λ; and from that find the conditions in which the solutions hold, giving us:
zi(λ) ∗ =
1
2λ ·
{ vi − β vi > β
0 otherwise vi + β vi < β . (17)
This can also be compactly written as:
z(λ)∗ = 1
2λ z̃, (18)
z̃ = s2 v − βs
where s = sign(z∗) ∈ {−1, 0, 1}|z| and s2 = s s ∈ {0, 1}|z|. The sign vector of z∗ can be determined without knowing λ, as λ is a Lagrangian multiplier for an inequality it must be nonnegative and therefore does not change the sign of the optimal solution. Lastly, we define the squared `2-norm of z̃, a result that will be used later:
‖z̃‖22 = z̃ᵀ(s2 v)− βz̃ᵀs = z̃ᵀv − β‖z̃‖1. (19)
Substituting z(λ)∗ back into the Lagrangian we get:
L(z(λ)∗, λ) = 1 2λ vᵀz̃− β 2λ ‖z̃‖1 + λ
( 1− 1
4λ2 ‖z̃‖22
) ,
and the derivative w.r.t. λ is:
∂L(z(λ)∗
∂λ = − 1 2λ2 vᵀz̃ + β 2λ2 ‖z̃‖1 + 1 + 1 4λ2 ‖z̃‖22.
Setting the derivative equal to zero and using the result from Eq. 19, we can find the optimal solution to λ:
λ2 = 1 2 z̃ᵀv − β 2 ‖z̃‖1 − 1 4 ‖z̃‖22 = 1 2 ‖z̃‖22 − 1 4 ‖z̃‖22
=⇒ λ∗ = 1 2 ‖z̃‖2.
Finally, plugging λ∗ into Eq. 18 we find the optimal solution
z∗ = z̃
‖z̃‖2 . (20)

First, I'd like to thank the authors for their answers and clarifications.
I find, the presentation of the multi-stage version of the model much clearer now.
Pros:
+ The paper states a sparse coding problem using cosine loss, which allows to solve the problem in a single pass.
+ The energy-based formulation allows bi-directional coding that incorporates top-down and bottom-up information in the feature extraction process.
Cons:
+ The cost of running the evaluation could be large in the  multi-class setting, rendering the approach less attractive and the computational cost comparable to recurrent architectures.
+ While the model is competitive and improves over the baseline, the paper would be more convincing with other comparisons (see text). The experimental evaluation is limited (a single database and a single baseline)
------
The motivation of the sparse coding scheme is to perform inference in a feed forward manner. This property does not hold in the multi stage setting, thus optimization would be required (as clarified by the authors).
Having an efficient way of performing a bi-directional coding scheme is very interesting. As the authors clarified, this could not necessarily be the case, as the model needs to be evaluated many times for performing a classification.
Maybe an interesting combination would be to run the model without any class-specific bias, and evaluation only the top K predictions with the energy-based setting.
Having said this, it would be good to include a discussion (if not direct comparisons) of the trade-offs of using a model as the one proposed by Cao et al. Eg. computational costs, performance.
Using the bidirectional coding only on the top layers seems reasonable: one can get a good low level representation in a class agnostic way. This, however could be studied in more detail, for instance showing empirically the trade offs. If I understand correctly, now only one setting is being reported.
Finally, the authors mention that one benefit of using the architecture derived from the proposed coding method is the spherical normalization scheme, which can lead to smoother optimization dynamics. Does the baseline (or model) use batch-normalization? If not, seems relevant to test.
Minor comments:
I find figure 2 (d) confusing. I would not plot this setting as it does not lead to a function (as the authors state in the text).


Despite recent advances, memory-augmented deep neural networks are still limited when it comes to life-long and one-shot learning, especially in remembering rare events. We present a large-scale life-long memory module for use in deep learning. The module exploits fast nearest-neighbor algorithms for efficiency and thus scales to large memory sizes. Except for the nearest-neighbor query, the module is fully differentiable and trained end-to-end with no extra supervision. It operates in a life-long manner, i.e., without the need to reset it during training. Our memory module can be easily added to any part of a supervised neural network. To show its versatility we add it to a number of networks, from simple convolutional ones tested on image classification to deep sequence-to-sequence and recurrent-convolutional models. In all cases, the enhanced network gains the ability to remember and do life-long one-shot learning. Our module remembers training examples shown many thousands of steps in the past and it can successfully generalize from them. We set new state-of-the-art for one-shot learning on the Omniglot dataset and demonstrate, for the first time, life-long one-shot learning in recurrent neural networks on a large-scale machine translation task.
1 INTRODUCTION
Machine learning systems have been successful in many domains, from computer vision (Krizhevsky et al., 2012) to speech recognition (Hinton et al., 2012) and machine translation (Sutskever et al., 2014; Bahdanau et al., 2014; Cho et al., 2014). Neural machine translation (NMT) is so successful that for some language pairs it approaches, on average, the quality of human translators (Wu et al., 2016). The words on average are crucial though. When a sentence resembles one from the abundant training data, the translation will be accurate. However, when encountering a rare word such as Dostoevsky (in German, Dostojewski), many models will fail. The correct German translation of Dostoevsky does not appear enough times in the training data for the model to sufficiently learn its translation.
While more example sentences concerning the famous Russian author might eventually be added to the training data, there are many other rare words or rare events of other kinds. This illustrates a general problem with current deep learning models: it is necessary to extend the training data and re-train them to handle such rare or new events. Humans, on the other hand, learn in a life-long fashion, often from single examples.
We present a life-long memory module that enables one-shot learning in a variety of neural networks. Our memory module consists of key-value pairs. Keys are activations of a chosen layer of a neural network, and values are the ground-truth targets for the given example. This way, as the network is trained, its memory increases and becomes more useful. Eventually it can give predictions that
∗First two authors contributed equally. †Work done as a member of the Google Brain Residency program (g.co/brainresidency). ‡Work done during internship at Google Brain.
leverage on knowledge from past data with similar activations. Given a new example, the network writes it to memory and is able to use it afterwards, even if the example was presented just once.
There are many advantages of having a long-term memory. One-shot learning is a desirable property in its own right, and some tasks, as we will show below, are simply not solvable without it. Even real-world tasks where we have large training sets, such as translation, can benefit from long-term memory. Finally, since the memory can be traced back to training examples, it might help explain the decisions that the model is making and thus improve understandability of the model.
It is not immediately clear how to measure the performance of a life-long one-shot learning model, since most deep learning evaluations focus on the average performance and do not have a one-shot component. We therefore evaluate in a few ways, to show that our memory module indeed works:
(1) We evaluate on the well-known one-shot learning task Omniglot, which is the only dataset with explicit one-shot learning evaluation. This dataset is small and does not benefit from life-long learning capability of our module, but we still exceed the best previous results and set new state-of-the-art.
(2) We devise a synthetic task that requires life-long one-shot learning. On this task, standard models fare poorly while our model can solve it well, demonstrating its strengths.
(3) Finally, we train an English-German translation model that has our life-long one-shot learning module. It retains very good performance on average and is also capable of one-shot learning. On the qualitative side, we find that it can translate rarely-occurring words like Dostoevsky. On the quantitative side, we see that the BLEU score for the generated translations can be significantly increased by showing it related translations before evaluating.
2 MEMORY MODULE
Our memory consists of a matrix K of memory keys, a vector V of memory values, and an additional vector A that tracks the age of items stored in memory. Keys can be arbitrary vectors of size key-size, and we assume that the memory values are single integers representing a class or token ID. We define a memory of size memory-size as a triple:
M = (Kmemory-size×key-size, Vmemory-size, Amemory-size).
A memory query is a vector of size key-size which we assume to be normalized, i.e., ‖q‖ = 1. Given a query q, we define the nearest neighbor of q inM as any of the keys that maximize the dot product with q: NN(q,M) = argmaxi q ·K[i]. Since the keys are normalized, the above notion corresponds to the nearest neighbor with respect to cosine similarity. We will also use the natural extension of it to k nearest neighbors, which we denote NNk(q,M). In our experiments we always used the set of k = 256 nearest neighbors. When given a query q, the memoryM = (K,V,A) will compute k nearest neighbors (sorted by decreasing cosine similarity): (n1, . . . , nk) = NNk(q,M) and return, as the main result, the value V [n1]. Additionally, we will compute the cosine similarities di = q ·K[ni] and return softmax(d1 · t, . . . , dk · t). The parameter t denotes the inverse of softmax temperature and we set it to t = 40 in our experiments. In models where the memory output is again embedded into a dense vector, we multiply the embedded output by the corresponding softmax component so as to provide a signal about confidence of the memory.
The forward computation of the memory module is thus very simple, the only interesting part being how to compute nearest neighbors efficiently, which we discuss below. But we must also answer the question how the memory is trained.
Memory Loss. Assume now that in addition to a query q we are also given the correct desired (supervised) value v. In the case of classification, this v would be the class label. In a sequenceto-sequence task, v would be the desired output token of the current time step. After computing the k nearest neighbors (n1, . . . , nk) as above, let p be the smallest index such that V [np] = v and
Case 1: V [n1] = v; Loss = [q · kb − q · k1 + α]+ Update: K[n1]← q+k1‖q+k1‖ A[n1]← 0
Case 2: V [n1] 6= v; Loss = [q · k1 − q · kp + α]+ Update: K[n′]← q V [n′]← v A[n′]← 0
b the smallest index such that V [nb] 6= v. We call np the positive neighbor and nb the negative neighbor. When no positive neighbor is among the top-k, we pick any vector from memory with value v instead of K[np]. We define the memory loss as:
loss(q, v,M) = [q ·K[nb]− q ·K[np] + α]+ .
Recall that both q and the keys in memory are normalized, so the products in the above loss term correspond to cosine similarities between q, the positive key, and the negative key. Since cosine similarity is maximal for equal terms, we want to maximize the similarity to the positive key and minimize the similarity to the negative one. But once they are far enough apart (by the margin α, 0.1 in all our experiments), we do not propagate any loss. This definition and reasoning behind it are almost identical to the one in Schroff et al. (2015) and similar to many other distance metric learning works (Weinberger & Saul, 2009; Weston et al., 2011).
Memory Update. In addition to computing the loss, we will also update the memoryM to account for the fact that the newly presented query q corresponds to v. The update is done in a different way depending on whether the main value returned by the memory module already is the correct value v or not. As before, let n1 = NN(q,M) be the nearest neighbor to q. If the memory already returns the correct value, i.e., if V [n1] = v, then we only update the key for n1 by taking the average of the current key and q and normalizing it:
K[n1]← q +K[n1]
‖q +K[n1]‖ .
When doing this, we also re-set the age: A[n1]← 0. Otherwise, when V [n1] 6= v, we find a new place in the memory and write the pair (q, v) there. Which place should we choose? We find memory items with maximum age, and write to one of those (randomly chosen). More formally, we pick n′ = argmaxiA[i] + ri where |ri| |M| is a random number that introduces some randomness in the choice so as to avoid race conditions in asynchronous multi-replica training. We then set:
K[n′]← q, V [n′]← v, A[n′]← 0.
With every memory update we also increment the age of all non-updated indices by 1. The full operation of the memory module is depicted in Figure 1.
Efficient nearest neighbor computation. The most expensive operation in our memory module is the computation of k nearest neighbors. This can be done exactly or in an approximate way.
In the exact mode, to calculate the nearest neighbors in K to a mini-batch of queries Q = (q1, . . . , qb), we perform a single matrix multiplication: Q×KT . This multiplies the batch-size
× key-size matrix Q by the key-size × memory-size matrix KT , and the result is the batch-size × memory-size matrix of all distances, from which we can choose the top-k. This procedure is linear in memory-size, so it can be expensive for very large memory sizes. But matrix multiplication is very heavily optimized, so in our experiments on GPUs we find that this operation is not a bottleneck for memory sizes up to half a million.
If the exact mode is too slow, the k nearest neighbors can be computed approximately using locality sensitive hashing (LSH). LSH is a hashing scheme so that near neighbors get similar hashes (Indyk & Motwani, 1998; Andoni & Indyk, 2006). For cosine similarity, the computation of an LSH is very simple. We pick a number of random normalized hash vectors h1, . . . , hl. The hash of a query q is a sequence of l bits, b1, . . . , bl, such that bi = 1 if, and only if, q · hi > 0. It turns out that near neighbors will, with high probability, have a large number of identical bits in their hash. To compute the nearest neighbors it is therefore sufficient to only look into parts of the memory with similar hashes. This makes the nearest neighbor computation work in approximately constant time – we only need to multiply the query by the hash vectors, and then only use the nearest buckets.
2.1 USING THE MEMORY MODULE
The memory module presented above can be added to any classification network. There are two main choices: which layer to use to generate queries, and how to use the output of the module.
In the simplest case, we use the final layer of a network as query and the output of the module is directly used for classification. This simplest case is similar to matching networks (Oriol Vinyals, 2016b) and our memory module yields good results already in this setting (see below).
Instead of using the output of the module directly, it is possible to embed it again into a dense representation and mix it with other predictions made by the network. To study this setting, we add the memory module to sequence-to-sequence recurrent neural networks. As described in detail below, a query to memory is made in every step of the decoder network. Memory output is embedded again into a dense representation and combined with inputs from other layers of the network.
Convolutional Network with Memory. To test our memory module in a simple setting, we first add it to a basic convolutional network network for image classification. Our network consists of two convolutional layers with ReLU non-linearity, followed by a max-pooling layer, another two convolutional-ReLU layers, another max-pooling, and two fully connected layers. All convolutions use 3× 3 filters with 64 channels in the first pair, and 128 in the second. The fully connected layers have dimension 256 and dropout applied between them. The output of the final layer is used as query to our memory module and the nearest neighbor returned by the memory is used as the final network prediction. Even this basic architecture yields good results in one-shot learning, as discussed below.
Sequence-to-sequence with Memory. For large-scale experiments, we add the memory module into a large sequence-to-sequence model. Such sequence-to-sequence recurrent neural networks (RNNs) with long short-term memory (LSTM) cells (Hochreiter & Schmidhuber, 1997) have proven especially successful at natural language processing (NLP) tasks, including machine translation (Sutskever et al., 2014; Bahdanau et al., 2014; Cho et al., 2014). We add the memory module to the Google Neural Machine Translation (GNMT) model (Wu et al., 2016). This model consists of an encoder RNN, which creates a representation of the source language sentence, and a decoder RNN that outputs the target language sentence. We left the encoder RNN unmodified. In the decoder RNN, we use the vector retrieved by the attention mechanism as query to the memory module. In the GNMT model, the attention vector is used in all LSTM layers beyond the second one, so the computation of the other layers and the memory can happen in parallel. Before the final softmax layer, we combine the embedded memory output with the output of the final LSTM layer using an additional linear layer, as depicted in Figure 2.
Extended Neural GPU with Memory. To test versatility of our memory module, we also add it to the Extended Neural GPU, a convolutional-recurrent model introduced by Kaiser & Bengio (2016). The Extended Neural GPU is a sequence-to-sequence model too, but its decoder is convolutional and the size of its state changes depending on the size of the input. Again, we leave the encoder part of the model intact, and extend the decoder part by a memory query. This time, we use the position one step ahead to query memory, and we put the embedded result to the output tape, as shown in Figure 3. Note that in this model the result of the memory will be processed by two recurrent-convolutional cells before the corresponding output is produced. The fact that this model still does one-shot learning confirms that the output of our memory module can be used deep inside a network, not just near the output layer.
3 RELATED WORK
Memory in Neural Networks. Augmenting neural networks with memory has been heavily studied recently. Many of these approaches design a memory component that is intended as a generalization of the memory in standard recurrent neural networks. In recurrent networks, the state passed from one time step to the next can be interpreted as the network’s memory representation of the current example. Moving away from this fixed-length vector representation of memory to a larger and more versatile form is at the core of these methods.
Augmenting recurrent neural networks with attention (Bahdanau et al., 2014) can be interpreted as creating a large memory component that allows content-based addressing. More generally, Graves et al. (2014) augmented a recurrent neural network with a computing-inspired memory component that can be addressed via both content- and address-based queries. Sukhbaatar et al. (2015) present a similar augmentation and show the importance of allowing multiple reads and writes to memory between inputs. These approaches excel at tasks where it is necessary to store large parts of a se-
quential input in a representation that can later be precisely queried. Such tasks include algorithmic sequence manipulation tasks, natural language modelling, and question-answering tasks.
The success of these approaches hinges on making the memory component fully differentiable and backpropagating signal through every access of memory. In this setting, computational requirements necessitate that the memory be small. Some attempts have been made at making hard access queries to memory (Zaremba & Sutskever, 2015; Xu et al., 2015), but it was usually challenging to match the soft version. Recently, more successful training for hard queries was reported (Gülçehre et al., 2016) that makes use of a curriculum strategy that mixes soft and hard queries at training time. Our approach applies hard access as well, but we encourage the model to make good queries via a special memory loss.
Modifications to allow for large-scale memory in neural networks have been proposed. The original implementation of memory networks (Weston et al., 2014) and later work on scaling it (Bordes et al., 2015; Chandar et al., 2016) used memory with size in the millions. The cost of doing so is that the memory must be fixed prior to training. Moreover, since during the beginning of training the model is unlikely to query the memory correctly, strong supervision is used to encourage the model to query memory locations that are useful. These hints are either given as additional supervising information by the task or determined heuristically as in Hill et al. (2015).
All the work discussed so far has either used a memory that is fixed before training or used a memory that is not persistent between different examples. For one-shot and lifelong learning, a memory must necessarily be both volatile during training and persistent between examples. To bridge this gap, Santoro et al. (2016) propose to partition training into distinct episodes consisting of a sequence of labelled examples {(xi, yi)}ni=1. A network augmented with a fully-differentiable memory is trained to predict yi given the previous sequence (x1, y1, . . . , xi−1). This way, the model learns to store important examples with their corresponding labels in memory and later re-use this information to correctly classify new examples. This model successfully exhibits one-shot learning on Omniglot.
However, this approach again requires fully-differentiable memory access and thus limits the size of the memory as well as the length of an episode. This restriction has recently been alleviated by Rae et al. (2016). Their model can utilize large memories, but unlike our work does not have an explicit cost to guide the formation of memory keys.
For classification tasks like Omniglot, it is easy to construct short episodes so that they include a few examples from each of several classes. However, this becomes harder as the output becomes richer. For example, in the difficult sequence-to-sequence tasks which we consider, it is hard to determine which examples would be helpful for correctly predicting others a priori, and so constructing short episodes each containing examples that are similar and act as hints to each other is intractable.
One-shot Learning. While the recent work of Santoro et al. (2016) succeeded in bridging the gap between memory-based models and one-shot learning, the field of one-shot learning has seen a variety of different approaches over time.
Early work utilized Bayesian methods to model data generatively (Fei-Fei et al., 2006; Lake et al., 2011). The paper that introduced the Omniglot dataset (Lake et al., 2011) approached the task with a generative model for strokes. This way, given a single character image, the probability of a different image being of the same character may be approximated via standard techniques. One early neural network approach to one-shot learning was given by Siamese networks (Koch, 2015). When our approach is applied to the Omniglot image classification dataset, the resulting training algorithm is actually similar to that of Siamese networks. The only difference is in the loss function: Siamese networks utilize a cross-entropy loss whereas our method uses a margin triplet loss.
A more sophisticated neural network approach is given by Vinyals et al. (2016). The strengths of this approach are (1) the model architecture utilizes recent advances in attention-augmented neural networks for set-to-set learning (Oriol Vinyals, 2016a), and (2) the training algorithm is designed to exactly match the testing phase (given k distinct images and an additional image, the model must predict which of the k images is of the same class as the additional image). This approach may also be considered as a generalization of previous work on metric learning.
4 EXPERIMENTS
We perform experiments using all three architectures described above. We experiment both on realworld data and on synthetic tasks that give us some insight into the performance and limitations of the memory module. In all our experiments we use the Adam optimizer (Kingma & Ba, 2014) and the parameters for the memory module remain unchanged (k = 256, α = 0.1). Good performance with a single set of parameters shows the versatility of our memory module. The source code for the memory module, together with our settings for Omniglot, is available on github1.
Omniglot. The Omniglot dataset (Lake et al., 2011) consists of 1623 characters from 50 different alphabets, each hand-drawn by 20 different people. The large number of classes (characters) with relatively few data per class (20), makes this an ideal data set for testing one-shot classification. In the N -way Omniglot task setup we pick N unseen character classes, independent of alphabet. We provide the model with one drawing of each character and measure its accuracy theK-th time it sees the character class. Our setup is identical to Oriol Vinyals (2016b), so we also augmented the data set with random rotations by multiples of 90 degrees and use 1200 characters for training, and the remaining character classes for evaluation. We present the results from Oriol Vinyals (2016b) and ours in Table 1. Even with a simpler network without batch normalization, we get similar results.
Synthetic task. To better understand the memory module operation and to test what it can remember, we devise a synthetic task and train the Extended Neural GPU with and without memory (we use a small Extended Neural GPU with 32 channels and memory of size half a million).
To create training and test data for our synthetic task, we use symbols from the set S = {2, . . . , 16000} and first fix a random function f : S → S. The function f is chosen at random, but fixed and the same for all training and testing examples (we used 40K training examples).
In our synthetic task, the input is a sequence consisting of As and Bs with one continuous substring of 7 digits from the set 0, 1, 2, 3. The substring is interpreted as a number written in base-4, e.g., 1982 = 1323324, so the string 132332 would be interpreted as 1982. The corresponding output is created by copying all As and Bs, but mapping the number through the random function f . For instance, assuming f(1982) = 3726, the output corresponding to 132332 would be 322032 as 3726 = 3220324. Here is an example of an input-output pair:
Input A 0 1 3 2 3 3 2 B A B A B Output A 0 3 2 2 0 3 2 B A B A B
This task clearly requires memory to store the fixed random function. Since there are 16K elements to learn, it is hard to memorize, and each single instance occurs quite rarely. The raw Extended Neural GPU (or any other sequence-to-sequence model) are limited by their size. With long training, the small model can memorize some of the sequences, but it is only a small fraction.
Additionally, there is no direct indication in the data what part of the input should trigger the production of each output symbol. For example, to produce the first 3 output in the above example, the
1 https://github.com/tensorflow/models/tree/master/learning_to_remember_ rare_events
memory key needs to encode all base-4 symbols from the input. Not just one or two aligned symbols, but a number of them. Moreover, it should not encode more symbols or it will not generalize to the test set. Similarly, a basic nearest neighbor classifier fails on this task. We use sequences of length up to 40 during training, but there are only 7 relevant symbols. The simple nearest neighbor by Hamming distance will most probably select some sequence with similar prefix or suffix of As and Bs, and not the one with the corresponding base-4 part. We also trained a large sequence-tosequence model with attention on this task (a 2-layer LSTM model with 256 units in each layer). This model can memorize the whole training set, but it suffers from a similar problem as the Hamming nearest neighbor – it almost doesn’t generalize, its accuracy on the test set is only about 1%. The same model with a memory module generalizes much better, reaching over 30% accuracy. The Extended Neural GPU with our memory module yields even better results, see Table 2.
Translation. To evaluate the memory module in a large-scale setting we use the GNMT model (Wu et al., 2016) extended with our memory module on the WMT14 English-to-German translation task. We evaluate the model both qualitatively and quantitatively.
On the qualitative side, we note that our memory-augmented model can successfully translate rare words like Dostoevsky, unlike the baseline model which predicts an identity-mapped Dostoevsky for the German translation of Dostoevsky.
On the quantitative side, we use the WMT test set. We find that in terms of BLEU score, an aggregate measure, the memory-augmented GNMT is on par with the baseline GNMT, see Table 3.
To evaluate our memory-augmented model for one-shot capabilities we split the test set in two. We take the even lines of the test set (index starting at 0) as a context set and the odd lines of the test set as the one-shot evaluation set. While showing the context set to the model, no additional training occurs, only memory updates are allowed. So the weights of the model do not change, but the memory does. Since the sentences in the test set are highly-correlated to each other (they come from paragraphs with preserved order), we expect that if we allow a one-shot capable model to use the context set to update its memory and then evaluate it on the other half of the test set, its accuracy will increase. For our GNMT with memory model, we passed the context set through the memory update operations 3 times. As seen in Table 3, the context set indeed helps when evaluating on the odd lines, increasing the BLEU score by almost 0.5. As further indication that our memory module works properly, we also evaluate the model after showing the whole test set as a context set. Note that this is essentially an oracle: the memory module gets to see all the correct answers, we do this only to test and debug. As expected, this increases BLEU score dramatically, by over 8 points.
5 DISCUSSION
We presented a long-term memory module that can be used for life-long learning. It is versatile, so it can be added to different deep learning models and at different layers to give the networks one-shot learning capability. Several parts of the presented memory module could be tuned and studied in more detail. The update rule that averages the query with the correct key could be parametrized. Instead of returning only the single nearest neighbor we could also return a number of them to be processed by other layers of the network. We leave these questions for future research.
The main issue we encountered, though, is that evaluating one-shot learning is difficult, as standard metrics do not focus on this scenario. In this work, we adapted the standard metrics to investigate our approach. For example, in the translation task we used half of the test set as context for the other half, and we still report the standard BLEU score. This allows us to show that our module works, but it is only a temporary solution. Better metrics are needed to accelerate progress of one-shot and life-long learning. Thus, we consider the present work as just a first step on the way to making deep models learn to remember rare events through their lifetime.

This paper proposes a new memory module for large scale life-long and one-shot learning. The module is general enough that the authors apply the module to several neural network architectures and show improvements in performance.
Using k-nearest neighbors for memory access is not completely new. This has been recently explored in Rae et al., 2016 and Chandar et al., 2016. K-nearest neighbors based memory for one-shot learning has also been explored in [R1]. This paper provides experimental evidence that such an approach can be applied to a variety of architectures.
Authors have addressed all my pre-review questions and I am ok with their response.
Are the authors willing to release the source code to reproduce the results? At least for omniglot experiments and synthetic task experiments?
References:
[R1] Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z. Leibo, Jack Rae, Daan Wierstra, Demis Hassabis: Model-Free Episodic Control. CoRR abs/1606.04460 (2016)


Recent papers have shown that neural networks obtain state-of-the-art performance on several different sequence tagging tasks. One appealing property of such systems is their generality, as excellent performance can be achieved with a unified architecture and without task-specific feature engineering. However, it is unclear if such systems can be used for tasks without large amounts of training data. In this paper we explore the problem of transfer learning for neural sequence taggers, where a source task with plentiful annotations (e.g., POS tagging on Penn Treebank) is used to improve performance on a target task with fewer available annotations (e.g., POS tagging for microblogs). We examine the effects of transfer learning for deep hierarchical recurrent networks across domains, applications, and languages, and show that significant improvement can often be obtained. These improvements lead to improvements over the current state-ofthe-art on several well-studied tasks.1
1 INTRODUCTION
Sequence tagging is an important problem in natural language processing, which has wide applications including part-of-speech (POS) tagging, text chunking, and named entity recognition (NER). Given a sequence of words, sequence tagging aims to predict a linguistic tag for each word such as the POS tag.
An important challenge for sequence tagging is how to transfer knowledge from one task to another, which is often referred to as transfer learning (Pan & Yang, 2010). Transfer learning can be used in several settings, notably for low-resource languages (Zirikly & Hagiwara, 2015; Wang & Manning, 2014) and low-resource domains such as biomedical corpora (Kim et al., 2003) and Twitter corpora (Ritter et al., 2011)). In these cases, transfer learning can improve performance by taking advantage of more plentiful labels from related tasks. Even on datasets with relatively abundant labels, multi-task transfer can sometimes achieve improvement over state-of-the-art results (Collobert et al., 2011).
Recently, a number of approaches based on deep neural networks have addressed the problem of sequence tagging in an end-to-end manner (Collobert et al., 2011; Lample et al., 2016; Ling et al., 2015; Ma & Hovy, 2016). These neural networks consist of multiple layers of neurons organized in a hierarchy and can transform the input tokens to the output labels without explicit hand-engineered feature extraction. The aforementioned neural networks require minimal assumptions about the task at hand and thus demonstrate significant generality—one single model can be applied to multiple applications in multiple languages without changing the architecture. A natural question is whether the representation learned from one task can be useful for another task. In other words, is there a way we can exploit the generality of neural networks to improve task performance by sharing model parameters and feature representations with another task?
To address the above question, we study the transfer learning setting, which aims to improve the performance on a target task by joint training with a source task. We present a transfer learning approach based on a deep hierarchical recurrent neural network, which shares the hidden feature repre-
1Code is available at https://github.com/kimiyoung/transfer
sentation and part of the model parameters between the source task and the target task. Our approach combines the objectives of the two tasks and uses gradient-based methods for efficient training. We study cross-domain, cross-application, and cross-lingual transfer, and present a parameter-sharing architecture for each case. Experimental results show that our approach can significantly improve the performance of the target task when the the target task has few labels and is more related to the source task. Furthermore, we show that transfer learning can improve performance over state-ofthe-art results even if the amount of labels is relatively abundant.
We have novel contributions in two folds. First, our work is, to the best of our knowledge, the first that focuses on studying the transferability of different layers of representations for hierarchical RNNs. Second, different from previous transfer learning methods that usually focus on one specific transfer setting, our framework exploits different levels of representation sharing and provides a unified framework to handle cross-application, cross-lingual, and cross-domain transfer.
2 RELATED WORK
There are two common paradigms for transfer learning for natural language processing (NLP) tasks, resource-based transfer and model-based transfer. Resource-based transfer utilizes additional linguistic annotations as weak supervision for transfer learning, such as cross-lingual dictionaries (Zirikly & Hagiwara, 2015), corpora (Wang & Manning, 2014), and word alignments (Yarowsky et al., 2001). Resource-based methods demonstrate considerable success in cross-lingual transfer, but are quite sensitive to the scale and quality of the additional resources. Resource-based transfer is mostly limited to cross-lingual transfer in previous works, and there is not extensive research on extending resource-based methods to cross-domain and cross-application settings.
Model-based transfer, on the other hand, does not require additional resources. Model-based transfer exploits the similarity and relatedness between the source task and the target task by adaptively modifying the model architectures, training algorithms, or feature representation. For example, Ando & Zhang (2005) proposed a transfer learning framework that shares structural parameters across multiple tasks, and improve the performance on various tasks including NER; Collobert et al. (2011) presented a task-independent convolutional neural network and employed joint training to transfer knowledge from NER and POS tagging to chunking; Peng & Dredze (2016) studied transfer learning between named entity recognition and word segmentation in Chinese based on recurrent neural networks. Cross-domain transfer, or domain adaptation, is also a well-studied branch of model-based transfer in NLP. Techniques in cross-domain transfer include the design of robust feature representations (Schnabel & Schütze, 2014), co-training (Chen et al., 2011), hierarchical Bayesian prior (Finkel & Manning, 2009), and canonical component analysis (Kim et al., 2015).
While our approach falls into the paradigm of model-based transfer, in contrast to the above methods, our method focuses on exploiting the generality of deep recurrent neural networks and is applicable to transfer between domains, applications, and languages.
Our work builds on previous work on sequence tagging based on deep neural networks. Collobert et al. (2011) develop end-to-end neural networks for sequence tagging without hand-engineered features. Later architectures based on different combinations of convolutional networks and recurrent networks have achieved state-of-the-art results on many tasks (Collobert et al., 2011; Huang et al., 2015; Chiu & Nichols, 2015; Lample et al., 2016; Ma & Hovy, 2016). These models demonstrate significant generality since they can be applied to multiple applications in multiple languages with a unified network architecture and without task-specific feature extraction.
3 APPROACH
In this section, we introduce our transfer learning approach. We first introduce an abstract framework for neural sequence tagging, summarizing previous work, and then discuss three different transfer learning architectures.
3.1 BASE MODEL
Though many different variants of neural networks have been proposed for the problem of sequence tagging, we find that most of the models can be described with the hierarchical framework illustrated in Figure 1(a). A character-level layer takes a sequence of characters (represented as embeddings) as input, and outputs a representation that encodes the morphological information at the character level. A word-level layer subsequently combines the character-level feature representation and a word embedding, and further incorporates the contextual information to output a new feature representation. After two levels of feature extraction (encoding), the feature representation output by the word-level layer is fed to a conditional random field (CRF) layer that outputs the label sequence.
Both of the word-level layer and the character-level layer can be implemented as convolutional neural networks (CNNs) or recurrent neural networks (RNNs) (Collobert et al., 2011; Chiu & Nichols, 2015; Lample et al., 2016; Ma & Hovy, 2016). We discuss the details of the model we use in this work in Section 3.4.
3.2 TRANSFER LEARNING ARCHITECTURES
We develop three architectures for transfer learning, T-A, T-B, and T-C, are illustrated in Figures 1(b), 1(c), and 1(d) respectively. The three architectures are all extensions of the base model discussed in the previous section with different parameter sharing schemes. We now discuss the use cases for the different architectures.
3.2.1 CROSS-DOMAIN TRANSFER
Since different domains are “sub-languages” that have domain-specific regularities, sequence taggers trained on one domain might not have optimal performance on another domain. The goal of cross-domain transfer is to learn a sequence tagger that transfers knowledge from a source domain to a target domain. We assume that few labels are available in the target domain.
There are two cases of cross-domain transfer. The two domains can have label sets that can be mapped to each other, or disparate label sets. For example, POS tags in the Genia biomedical corpus can be mapped to Penn Treebank tags (Barrett & Weber-Jahnke, 2014), while some POS tags in Twitter (e.g., “URL”) cannot be mapped to Penn Treebank tags (Ritter et al., 2011).
If the two domains have mappable label sets, we share all the model parameters and feature representation in the neural networks, including the word and character embedding, the word-level layer, the character-level layer, and the CRF layer. We perform a label mapping step on top of the CRF layer. This becomes the model T-A as shown in Figure 1(b).
If the two domains have disparate label sets, we untie the parameter sharing in the CRF layer—i.e., each task learns a separate CRF layer. This parameter sharing scheme reduces to model T-B as shown in Figure 1(c).
3.2.2 CROSS-APPLICATION TRANSFER
Sequence tagging has a couple of applications including POS tagging, chunking, and named entity recognition. Similar to the motivation in (Collobert et al., 2011), it is usually desirable to exploit the underlying similarities and regularities of different applications, and improve the performance of one application via joint training with another. Moreover, transfer between multiple applications can be helpful when the labels are limited.
In the cross-application setting, we assume that multiple applications are in the same language. Since different applications share the same alphabet, the case is similar to cross-domain transfer with disparate label sets. We adopt the architecture of model T-B for cross-application transfer learning where only the CRF layers are disjoint for different applications.
3.2.3 CROSS-LINGUAL TRANSFER
Though cross-lingual transfer is usually accomplished with additional multi-lingual resources, these methods are sensitive to the size and quality of the additional resources (Yarowsky et al., 2001; Wang & Manning, 2014). In this work, instead, we explore a complementary method that exploits the cross-lingual regularities purely on the model level.
Our approach focuses on transfer learning between languages with similar alphabets, such as English and Spanish, since it is very difficult for transfer learning between languages with disparate alphabets (e.g., English and Chinese) to work without additional resources (Zirikly & Hagiwara, 2015).
Model-level transfer learning is achieved through exploiting the morphologies shared by the two languages. For example, “Canada” in English and “Canadá” in Spanish refer to the same named entity, and the morphological similarities can be leveraged for NER and also POS tagging with nouns. Thus we share the character embeddings and the character-level layer between different languages for transfer learning, which is illustrated as the model T-C in Figure 1(d).
3.3 TRAINING
In the above sections, we introduced three neural architectures with different parameter sharing schemes, designed for different transfer learning settings. Now we describe how we train the neural networks jointly for two tasks.
Suppose we are transferring from a source task s to a target task t, with the training instances being Xs and Xt. Let Ws and Wt denote the set of model parameters for the source and target tasks respectively. The model parameters are divided into two sets, task specific parameters and shared parameters, i.e.,
Ws =Ws,spec ∪Wshared,Wt =Wt,spec ∪Wshared,
where shared parameters Wshared are jointly optimized by the two tasks, while task specific parameters Ws,spec and Wt,spec are trained for each task separately.
The training procedure is as follows. At each iteration, we sample a task (i.e., either s or t) from {s, t} based on a binomial distribution (the binomial probability is set as a hyperparameter). Given the sampled task, we sample a batch of training instances from the given task, and then perform a gradient update according to the loss function of the given task. We update both the shared parameters and the task specific parameters. We repeat the above iterations until stopping. We adopt AdaGrad (Duchi et al., 2011) to dynamically compute the learning rates for each iteration. Since the source and target tasks might have different convergence rates, we do early stopping on the target task performance.
3.4 MODEL IMPLEMENTATION
In this section, we describe our implementation of the base model. Both the character-level and word-level neural networks are implemented as RNNs. More specifically, we employ gated recurrent units (GRUs) (Cho et al., 2014). Let (x1,x2, · · · ,xT ) be a sequence of inputs that can be embeddings or hidden states of other layers. Let ht be the GRU hidden state at time step t. Formally, a GRU unit at time step t can be expressed as
rt = σ(Wrxxt +Wrhht−1)
zt = σ(Wzxxt +Wzhht−1)
h̃t = tanh(Whxxt +Whh(rt ht−1)) ht = zt ht−1 + (1− zt) h̃t,
where W ’s are model parameters of each unit, h̃t is a candidate hidden state that is used to compute ht, σ is an element-wise sigmoid logistic function defined as σ(x) = 1/(1 + e−x), and denotes element-wise multiplication of two vectors. Intuitively, the update gate zt controls how much the unit updates its hidden state, and the reset gate rt determines how much information from the previous hidden state needs to be reset. The input to the character-level GRUs is character embeddings, while the input to the word-level GRUs is the concatenation of character-level GRU hidden states and word embeddings. Both GRUs are bi-directional and have two layers.
Given an input sequence of words, the word-level GRUs and the character-level GRUs together learn a feature representation ht for the t-th word in the sequence, which forms a sequence h = (h1,h2, · · · ,hT ). Let y = (y1, y2, · · · , yT ) denote the tag sequence. Given the feature representation h and the tag sequence y for each training instance, the CRF layer defines the objective function to maximize based on a max-margin principle (Gimpel & Smith, 2010) as:
f(h,y)− log ∑
y′∈Y(h)
exp(f(h,y′) + cost(y,y′)),
where f is a function that assigns a score for each pair of h and y, and Y(h) denotes the space of tag sequences for h. The cost function cost(y,y′) is added based on the max-margin principle (Gimpel & Smith, 2010) that high-cost tags y′ should be penalized more heavily.
Our base model is similar to Lample et al. (2016), but in contrast to their model, we employ GRUs for the character-level and word-level networks instead of Long Short-Term Memory (LSTM) units, and define the objective function based on the max-margin principle. We note that our transfer learning framework does not make assumptions about specific model implementation, and could be applied to other neural architectures (Collobert et al., 2011; Chiu & Nichols, 2015; Lample et al., 2016; Ma & Hovy, 2016) as well.
4 EXPERIMENTS
4.1 DATASETS
We use the following benchmark datasets in our experiments: Penn Treebank (PTB) POS tagging, CoNLL 2000 chunking, CoNLL 2003 English NER, CoNLL 2002 Dutch NER, CoNLL 2002 Spanish NER, the Genia biomedical corpus (Kim et al., 2003), and a Twitter corpus (Ritter et al., 2011).
The statistics of the datasets are described in Table 1. We construct the POS tagging dataset with the instructions described in Toutanova et al. (2003). Note that as a standard practice, the POS tags are extracted from the parsed trees. For the CoNLL 2003 English NER dataset, we follow previous works (Collobert et al., 2011) to append one-hot gazetteer features to the input of the CRF layer for fair comparison. Since there is no standard training/dev/test data split for the Genia and Twitter corpora, we randomly sample 10% for test, 10% for development, and 80% for training. We follow previous work (Barrett & Weber-Jahnke, 2014) to map Genia POS tags to PTB POS tags.
4.2 TRANSFER LEARNING PERFORMANCE
We evaluate our transfer learning approach on the above datasets. We fix the hyperparameters for all the results reported in this section: we set the character embedding dimension at 25, the word embedding dimension at 50 for English and 64 for Spanish, the dimension of hidden states of the character-level GRUs at 80, the dimension of hidden states of the word-level GRUs at 300, and the initial learning rate at 0.01. Except for the Twitter datasets, these datasets are fairly large. To simulate a low-resource setting, we also use random subsets of the data. We vary the labeling rate of the target task at 0.001, 0.01, 0.1 and 1.0. Given a labeling rate r, we randomly sample a ratio r of the sentences from the training set and discard the rest of the training data—e.g., a labeling rate of 0.001 results in around 900 training tokens on PTB POS tagging (Cf. Table 1).
The results on transfer learning are plotted in Figure 2, where we compare the results with and without transfer learning under various labeling rates. The numbers in the y-axes are accuracies for POS tagging, and chunk-level F1 scores for chunking and NER. The numbers are shown in Table 2. We can see that our transfer learning approach consistently improved over the non-transfer results. We also observe that the improvement by transfer learning is more substantial when the labeling rate is lower. For cross-domain transfer, we obtained substantial improvement on the Genia and Twitter corpora by transferring the knowledge from PTB POS tagging and CoNLL 2003 NER. For example, as shown in Figure 2(a), we can obtain an tagging accuracy of 83%+ with zero labels and 92% with only 0.001 labels when transferring from PTB to Genia. As shown in Figures 2(d) and 2(e), our transfer learning approach can improve the performance on Twitter POS tagging and NER for all labeling rates, and the improvements with 0.1 labels are more than 8% for both datasets. Cross-application transfer also leads to substantial improvement under low-resource conditions. For
example, as shown in Figures 2(g) and 2(h), the improvements with 0.1 labels are 6% and 3% on CoNLL 2000 chunking and CoNLL 2003 NER respectively when transferring from PTB POS tagging. Figures 2(j) and 2(i) show that cross-lingual transfer can improve the performance when few labels are available.
Figure 2 further shows that the improvements by different architectures are in the following order: T-A > T-B > T-C. This phenomenon can be explained by the fact that T-A shares the most model parameters while T-C shares the least. Transfer settings like cross-lingual transfer can only use T-C because the underlying similarities between the source task and the target task are less prominent (i.e., less transferable), and in those cases the improvement by transfer learning is less substantial.
Another interesting comparison is among Figures 2(a), 2(b), and 2(c). Figure 2(a) is cross-domain transfer, Figure 2(b) is transfer across domains and applications at the same time, and Figure 2(c) combines all the three transfer settings (i.e., from Spanish NER in the general domain to English POS tagging in the biomedical domain). The results show that the improvement by transfer learning diminishes when the transfer becomes “indirect” (i.e., the source task and the target task are more loosely related).
We also study using different transfer learning models for the same task. We study the effects of using T-A, T-B, and T-C when transferring from PTB to Genia, and the results are included in the lower part of Table 2. We observe that the performance gain decreases when less parameters are shared (i.e., T-A > T-B > T-C).
4.3 COMPARISON WITH STATE-OF-THE-ART RESULTS
In the above section, we examine the effects of different transfer learning architectures. Now we compare our approach with state-of-the-art systems on these datasets.
We use publicly available pretrained word embeddings as initialization. On the English datasets, following previous works that are based on neural networks (Collobert et al., 2011; Huang et al., 2015; Chiu & Nichols, 2015; Ma & Hovy, 2016), we experiment with both the 50-dimensional SENNA embeddings (Collobert et al., 2011) and the 100-dimensional GloVe embeddings (Pennington et al., 2014) and use the development set to choose the embeddings for different tasks and settings. For Spanish and Dutch, we use the 64-dimensional Polyglot embeddings (Al-Rfou et al., 2013). We set the hidden state dimensions to be 300 for the word-level GRU. The initial learning rate for AdaGrad is fixed at 0.01. We use the development set to tune the other hyperparameters of our model.
Our results are reported in Table 3. Since there are no standard data splits on the Genia and Twitter corpora, we do not include these datasets into our comparison. The results for CoNLL 2000 chunking, CoNLL 2003 NER, and PTB POS tagging are obtained by transfer learning between the three tasks, i.e., transferring from two tasks to the other. The results for Spanish and Dutch NER are obtained with transfer learning between the NER datasets in three languages (English, Spanish, and Dutch). From Table 3, we can draw two conclusions. First, our transfer learning approach achieves new state-of-the-art results on all the considered benchmark datasets except PTB POS tagging, which indicates that transfer learning can still improve the performance even on datasets with
relatively abundant labels. Second, our base model (w/o transfer) performs competitively compared to the state-of-the-art systems, which means that the improvements shown in Section 4.2 are obtained over a strong baseline.
5 CONCLUSION
In this paper we develop a transfer learning approach for sequence tagging, which exploits the generality demonstrated by deep neural networks in previous work. We design three neural network architectures for the settings of cross-domain, cross-application, and cross-lingual transfer. Our transfer learning approach achieves significant improvement on various datasets under low-resource conditions, as well as new state-of-the-art results on some of the benchmarks. With thorough experiments, we observe that the following factors are crucial for the performance of our transfer learning approach: a) label abundance for the target task, b) relatedness between the source and target tasks, and c) the number of parameters that can be shared. In the future, it will be interesting to combine model-based transfer (as in this work) with resource-based transfer for cross-lingual transfer learning.
ACKNOWLEDGMENTS
This work was funded by NVIDIA, the Office of Naval Research grant N000141512791, the ADeLAIDE grant FA8750-16C-0130-001, the NSF grant IIS1250956, and Google Research.

Authors' response well answered my questions. Thanks!
Evaluation not changed.
###
This paper proposes a hierarchical framework of transfer learning for sequence tagging, which is expected to help the target task with the source task, by sharing as many levels of representation as possible. It is a general framework for various neural models. The paper has extensive and solid experiments, and the performance is competitive with the state of the art on multiple benchmark datasets. The framework is clear by itself, except that more details about training procedure, i.e. sec-3.3, need to be added.
The experimental results show that for some task pairs {s,t}, this framework can help low-resource target task t, and the improvement increases with more levels of representations can be shared. Firstly, I suggest that the terms *source* and *target* should be more precisely defined in the current framework, because, due to Sec-3.3, the s and t in each pair are sort of interchangeable. That is, either of them can be the *source* or *target* task, especially when p(X=s)=p(X=t)=0.5 is used in the task sampling. The difference is: one is low-resourced and the other is not. Thus it could be thought of as multi-tasking between tasks with imbalanced resource. So one question is: does this framework simultaneously help both tasks in the pair, by learning more generalizable representations for different domains/applications/languages? Or is it mostly likely to only help the low-resourced one? Does it come with sacrifice on the high-resourced side?
Secondly, as the paper shows that the low-resourced tasks are improved for the selected task pairs, it would also be interesting and helpful to know how often this could happen. That is, when the tasks are randomly paired (one chosen from a low-resource pool and the other from a high resource pool), how often could this framework help the low-resourced one?
Moreover, the choice of T-A/T-B/T-C lies intuitively in how many levels of representation *could* be shared as possible. This implicitly assumes share more, help more. Although I tend to believe so, it would be interesting to have some empirical comparison. For example, one could perhaps select some cross-domain pair, and see if T-A > T-B > T-C on such pairs, as mentioned in the author’s answer to the pre-review question.
In general, I think this is a solid paper, and more exploration could be done in this direction. So I tend to accept this paper.