PERCEPTION UPDATING NETWORKS: ON ARCHITEC- TURAL CONSTRAINTS FOR INTERPRETABLE VIDEO GENERATIVE MODELS
We investigate a neural network architecture and statistical framework that models frames in videos using principles inspired by computer graphics pipelines. The proposed model explicitly represents “sprites” or its percepts inferred from maximum likelihood of the scene and infers its movement independently of its content. We impose architectural constraints that forces resulting architecture to behave as a recurrent what-where prediction network.
1 INTRODUCTION
The current computer graphics pipelines are the result of efficient implementations required by limited hardware and high frequency output requirements. These requirements were also achieved with the use of explicit physics and optic constraints and modeling with constantly improving data structures (Shirley et al., 2015).
In machine learning on the other hand, for a long time image (Olshausen et al., 1996) and video (Hurri & Hyvärinen, 2003) generative models had been investigated with statistical approaches that model images down to the pixel level (Simoncelli & Olshausen, 2001), sometimes assuming neighborhood statistical dependencies (Osindero & Hinton, 2008). In video prediction, the current state of the art uses variations of deep convolutional recurrent neural networks (Kalchbrenner et al., 2016) (Lotter et al., 2016) (Finn et al., 2016).
As a parallel to the classic machine learning approach to image and video interpretation and prediction is a growing trend in the deep learning literature for modeling vision as inverse graphics (Kulkarni et al., 2015)(Rezende et al., 2016)(Eslami et al., 2016). These approaches can be interpreted into two groups: supervised and unsupervised vision as inverse graphics. The supervised approach assumes that during training an image is provided with extra information about its rotation, translation, illumination, etc. The goal of the supervised model is to learn an auto-encoder that explicitly factors out the content of the image and its physical properties. The supervised approach is illustrated by Kulkarni et al. (2015).
The unsupervised approach requires extra architectural constraints, similar to those assumed in computer graphics. For example, Reed et al. (2016) modeled the content of a scene with a Generative Adversarial Network (Goodfellow et al., 2014) and its location with Spatial Transformer Networks (Jaderberg et al., 2015). The full model is adapted end-to-end to generate images whose appearance can be changed by independently modifying the ”what” and/or ”where” variables. A similar approach was applied to video generation with volumetric convolutional neural networks (Vondrick et al., 2016).In two papers by Google DeepMind (Rezende et al., 2016) (Eslami et al., 2016) they improved the ”where” representations of the unsupervised approach and modeled the 3D geometry of the scene. This way they explicitly represented object rotation, translation, camera pose, etc. Their approaches were also trained end-to-end with REINFORCE-like stochastic gradients to backpropagate through non-differentiable parts of the graphics pipeline (Rezende et al., 2016) or to count
∗Companion code repo coming soon.
the number of objects in the scene (Eslami et al., 2016). Those papers also used Spatial Transformer Networks to model the position of the objects in the scene, but they extended it to 3D geometry so it could also model rotation and translation in a volumetric space.
Other approaches inspired by the graphics pipeline and computer vision geometry in machine learning uses the physics constraints to estimate the depth of each pixel in the scene and camera pose movements to predict frames in video (Mahjourian et al., 2016) (Godard et al., 2016).
The present paper is closer to the unsupervised approach of vision as inverse graphics. More precisely, here we investigate frame prediction in video. Contrary to the work by Reed et al. (2016) here we first limit ourselves to simple synthetic 2D datasets and learning models whose representations can be visually interpreted. This way we can investigate exactly what the neural network is learning and validate our statistical assumptions. Also, we investigate the behavior of Spatial Transformer Networks and question it as the default choice when limited compute resources are available and no scale invariance is required.
First in the next Section we will pose a statistical model that is appropriate for machine learning but inspired by the graphics pipeline.
2 A 2D STATISTICAL GRAPHICS PIPELINE
This section starts with a high level description of the 2D graphics pipeline, followed by a discussion of how to implement it with neural network modules, and finally we define a formal statistical model.
The 2D graphics pipeline starts from geometric primitives and follows with modeling transformations, clipping, viewing transformations and finally scan conversion for generating an image. Here, we will deal with previously rasterized bitmaps, i.e. sprites, and will model the translation transformations, rotation and clipping with differential operations. This way, the steps in the pipeline can be defined as layers of a neural network and the free parameters can be optimized with backpropagation.
For our neural network implementation, we assume a finite set of sprites (later we generalize it to infinite sprites) that will be part of the frames in the video. The image generation network selects a sprite, s, from a memorized sprite database Si∈{1,...,K} using an addressing signal c:
s = ∑ j
cjSj , where∑ j cj = 1. (1)
For interpretable results it would be optimal to do one-hot memory addressing where cj = 1 for Sj = S and cj = 0 otherwise. Note that (1) is differentiable w.r.t to both cj and Sj so we can learn the individual sprites from data. We can for cj to sum to 1 using the softmax nonlinearity. This approach was inspired by the recent deep learning literature on attention modules (Bahdanau et al., 2014) (Graves et al., 2014).
When the number of possible sprites is too large it is more efficient to do a compressed representation. Instead of using an address value c we use a content addressable memory where the image generator estimates a code z that is then decoded to the desired sprite with a (possibly nonlinear) function d(z). If we interpret the addressing value z as a latent representation and the content addressable memory d(z) as a decoder, we can use the recent advances in neural networks for generative models to setup our statistical model. We will revisit this later in this section.
The translation transformation can be modeled with a convolution with a Delta function or using spatial transformers. Note that the translation of an image I(x, y) can be defined as
I(x− τx, y − τy) = I(x, y) ? δ(x− τx, y − τy), (2)
where ? denotes the image convolution operation. Clipping is naturally handled in such a case. If the output images have finite dimensions and δ(x−τx, y−τy) is non-zero near its border, the translated image I(x− τx, y − τy) will be clipped. Another way of implementing the translation operation is using Spatial Transformer Networks (STN) (Jaderberg et al., 2015). An implementation of STN can be defined in two steps: resampling and bilinear interpolation. Resampling is defined by moving the position of the pixels (x, y) in the original image using a linear transform to new positions (x̃, ỹ) as
[ x̃ ỹ ] = A [ x y 1 ] , where
A = [ A11 A12 A13 A21 A22 A23 ] .
(3)
We assume the coordinates in the original image are integers 0 ≤ x < M and 0 ≤ y < N , where M ×N is the size of the image I . Once the new coordinates are defined, we can calculate the values of the pixels in the new image Ĩ using bilinear interpolation:
Ĩ(x̃, ỹ) = wx1,y1I(x1, y1) + wx1,y2I(x1, y2)+
wx2,y1I(x2, y1) + wx2,y2I(x2, y2) (4)
where (x1, x2, y1, y2) are integers, x1 ≤ x̃ < x2, y1 ≤ ỹ < y2 and
wx1,y1 = (bx̃c − x̃)(bỹc − x̃) wx1,y2 = (bx̃c − x̃)(bỹc+ 1− ỹ) wx2,y1 = (bx̃c+ 1− x̃)(bỹc − ỹ) wx2,y2 = (bx̃c − x̃)(bỹc+ 1− ỹ)
(5)
To avoid sampling from outside the image we clip the values bx̃c and bx̃c+1 between 0 and M and the values bỹc and bỹc + 1 between 0 and N . We omitted that in (5) for conciseness. Note that (4) is piecewise differentiable w.r.t I .
We can define translation through operations with
A = [ 1 0 τx 0 1 τy ] . (6)
Also, we can rotate the image ρ radians counter clockwise with
A = [ cos ρ sin ρ 0 − sin ρ cosρ 0 ] . (7)
Image rescaling is achieved on that framework by rescaling in the right square submatrix A1:2,1:2. We illustrate in Fig. 1 how to get similar results using convolutions with a delta-function and spatial transformers.
Considering the tools defined above, we can define a statistical model of 2D images the explicitly represents sprites and their positions in the scene. We can use the free energy of this statistical model to optimize a neural network. Let us start with a static single frame model and later generalize it to video.
Let an image I ∼ pθ(I) be composed of sprite s ∼ pθ(s) centered in the (x, y) coordinates in the larger image I . Denote these coordinates as a random variable δxy ∼ pθ, where θ are the model parameters. pθ(δxy) can be factored in two marginal categorical distributions Cat(δx) and Cat(δy) that models the probability of each coordinate of the sprite independently. For the finite sprite dataset, pθ(s) is also a categorical distribution conditioned on the true sprites. For this finite case the generative model can be factored as
pθ(I, s, δ) = pθ(s)pθ(δxy)p(I|s, δxy), (8)
assuming that “what”, s, and “where”, δxy , are statistically independent. Also, in such case the posterior
pθ(s, δ|I) = pθ(s|I)p(δxy|I) (9)
is tractable. One could use for instance Expectation-Maximization or greedy approaches like Matching Pursuit to alternate between the search for the position and fitting the best matching shape. For the infinite number of sprites case, we assume that there is a hidden variable z from which the sprites are generated as p(s, z) = pθ(z)pθ(s|z). In such case our full posterior becomes
pθ(z, s, δ|I) = pθ(z, s|I)p(δxy|I) = pθ(z|I)pθ(s|I, z)p(δxy|I).
(10)
We can simplify (10) assuming pθ(z|s) = pθ(z|I) for simple images without ambiguity and no sprite occlusion. For a scalable inference in the case of unknown θ and z and intractable pθ(z|s) we can use the auto-encoding variational Bayes (VAE) approach proposed by Kingma & Welling (2013). Using VAE we define an approximate recognition model qφ(z|s). In such case, the loglikelihood of the i.i.d images I is log pθ(I1, . . . , IT ) = ∑T i log pθ(Ii) and
log pθ(Ii) = DKL(qφ(z|si)||pθ(z|si))+ DKL(pθ(z|si)||pθ(z|Ii))+
L(θ, φ, δxy, Ii). (11)
Again, assume that the approximation pθ(z|s) = pθ(z|I) we have DKL(pθ(z|si)||pθ(z|Ii)) = 0 and the free energy (or variational lower bound) term equal to
L(θ, φ, δ, I) = −DKL(qφ(z|si)||pθ(z))+ Eqφ(z|s,δ)pθ(δ|I)[log pθ(I|z, δ)],
(12)
where we dropped the subindices xy and i to avoid clutter. Here we would like to train our model by maximizing the lower bound (12), again inspired by VAE. We can do so using the reparametrization trick assuming qφ(z|s) and the prior pθ(z) to be Gaussian and sampling
z = mφ(I) + vφ(I) · ξ, (13)
where ξ ∼ N (0, σI), I is the identity matrix, the functions m(I) and v(I) are deep neural networks learned from data.
One can argue that given z and a good approximation to the posterior qφ, estimating δ is still tractable. Nevertheless, we preemptively avoid Expectation-Maximization or other search approaches and use instead neural network layers lx and ly:
δxy = softmax(lx(I))⊗ softmax(ly(I)), (14)
with ⊗ denoting the outer product of marginals. We also experiment using STNs. Such amortized inference is also faster in training and test time than EM and will also cover the case where I is itself a learned low dimensional or latent representation instead of an observable image. Bear this in mind while we use this approach even in simple experiments such as those with moving shapes in the Experiments Section. This will help us to understand what can be learned from this model.
We extend the model above to videos, i.e. sequences of images I(t) = {I(0), I(1), . . .}, assuming that the conditional log-likelihood log pθ(It|HIt) = logpθ(It|Hδt , Hzt) follows (11), where HIt is the history of video frames prior to time point t. Also Hδt and Hzt are the history of position coordinates and the history of latent variables of the sprites respectively. We should observe that one can make the assumption that the sprites don’t change for a given video I(t) and only estimate one sprite st=0 or hidden variable zt=0. This assumption can be useful for long term predictions, but requires that the main object moving in the scene doesn’t change.
In the next section, we propose a neural network architecture for maximizing our approximate variational lower bound 2D videos.
3 PERCEPTION UPDATING NETWORKS
This Section proposes a family of neural architectures for optimizing the lower bound (12). A schematic diagram is represented in Fig. (2). The core of our method is a Recurrent Neural Network (RNN) augmented with task specific modules, namely a sprite addressable memory and modeling transformations layers. RNNs augmented with task specific units were popularized by Graves et al. (2014) in the context of learning simple differentiable algorithms and served as inspiration for us as well. Here since we explicitly model the perceived sprites as s or z and update it and its location and/or rotation though time we decided to call our method simply Perception Updating Networks.
Here an input frame at time t, It, is fed to the RNN that emits 2 signals: a memory address that selects a relevant sprite and transformation parameters. If we are doing the translation transformation using convolutions and delta functions this output is equal to (14). If using STN, the translation operation returns the matrix A used in (3). Note that we could use both, letting convolutions with δ to the translation is constraining A as in (7) to do rotation transformations only. We describe the general case where both δxy and STNs are used in Algorithm 1.
Beyond deciding between STNs vs δxy , a few other free parameters of our method are the type of RNN (e.g. vanilla RNN, LSTM, GRU, ConvRNN, etc), the number of neurons in the hidden state of the RNN and neural network architectures that infer the correct sprite and modeling transformation parameters. Our hyperparameter choices are investigated separately in each experiment in the next Section.
Data: input videos It, t ∈ {0, 1, 2, . . .}, initial RNN state h0, neural network layers mφ, vφ, d, l, f Result: video predictions It, t ∈ {1, 2, 3, . . .} for t ∈ {0, 1, . . .} do
ht ← RNN(It, ht−1) δxy = softmax(lx(ht))⊗ softmax(ly(ht)) ρ = f(ht)
A = [ cos ρ sin ρ 0 − sin ρ cos ρ 0 ] ξ ∼ pθ(z) zt = mφ(ht) + vφ(ht) · ξ st = d(zt) at = STN(st, A)
Ĩt+1 = at ? δxy It+1 = µĨt+1 + (1− µ)B
end Algorithm 1: Perception Updating Networks. STN denotes spatial transformer operator (3)-(4) and ? denotes convolution. We experimented with several variations of this algorithm, mainly changing if and how the “where” modules δxy and STN are used. Also changing how the sprite st is calculated and not using a background B when not necessary.
In the next section we present experiments with the proposed architecture on synthetic datasets.
4 EXPERIMENTS
In this section we experiment with several implementations of the proposed Perception Updating Networks. We start with a simple synthetic dataset made of videos where one of 3 moving shapes moves with constant speed bouncing in the edges of an image. This illustrates the working of the finite memory and the addressing scheme in (1). Afterwards we show results on the moving MNIST dataset (Srivastava et al., 2015) commonly used in the literature of generative neural network models of videos.
4.1 BOUNCING SHAPES
In this first experiment we generate videos of one of three shapes moving on a non-zero background. The shapes are a square, triangle and cross. The image size is 20×20 pixels and the shapes are 8×8 pixels. The pixel values are between 0 and 1. The shapes are picked with equal probability and they move at constant speed of 1 pixel per frame. The shapes start from random initial positions with and start moving in random directions as well.
We tested two implementations of the proposed architecture: one using only convolutions, referred to as convolutional PUN in the figures, and another using using spatial transformers, called spatial transformer PUN. For the parameters of the convolutional PUN the RNN used was a Long Short Term Memory (LSTM) with 100 cells. The RNN in the Spatial Transformer PUN had 256 cells. In the convolutional PUN, the location layers used to calculate δxy , lx and ly , output vectors of size 20 pixels and we used the finite addressable memory described in (1). The background is also learned from data as weights of neural network. This background served to make the task more difficult and force the network to avoid just exploiting any non-zero value. After the convolutional composition It = st ? δxy , we added the background to form a new image using Ĩt = µ · It + (1− µ)B, where µ is a differentiable mask that accounts for the “transparency” of the image It. B is the learned 20 × 20 pixels background image. For complex shapes this mask shape could be calculated as another module in the network, similarly to the approach in Vondrick et al. (2016).
In the following experiments, the training videos were 10 frames long. At test time the network is fed the first 10 frames of a video and asked to predict the next 10. Results for the compared methods are shown in Fig. ??. For the baseline method, we did a hyperparameter search on conventional LSTMs with a single linear output layer until we found one that had comparable results at test time. That network had 256 hidden cells. Also, note that although the scale of the mean square error is the same, the results from our proposed architecture look smoother than those learned by the LSTM as shown in Fig. 3.
Given such a simple experiment, it is elucidating to visualize values learned by each piece of the network. As expected the sprite memory learned the 3 investigated shapes in transposed order since they are reverted by the convolution operation to compose the frame. We also experimented with choosing the size of the learned sprites st smaller and larger than the true shapes. We observed that for larger shapes such as 10 × 10 the sprites converge to the correct shapes but just using part of the pixels. For smaller shapes such as 6 × 6 pixels, instead of learning a part of the correct shape, the convolutional Perception Updating Network learned to compensate for the lack of enough pixels with more than one non-zero value in the location operation δxy (see Fig. 3). This allow us to suggest to the interested practitioner that in order to get interpretable results it is better to use sprites larger than the expected size than smaller.
For the spatial transformer PUN the image is calculated as (see Algorithm 1 for context):
A = f(ht), It+1 = STN(st, A). (15)
We noticed that the spatial transformer PUN was not able to learn the training videos using an equivalent architecture to the convolutional PUN one. We had to use multiple layers to define the function f(ht). In other words, in the convolution based method δxy can be estimated by a single affine transformation of the state ht but A cannot. We also had to use smaller learning rates to
guarantee convergence: 0.0001 for STN while the δxy-based model worked with a value 10 times larger.
If we don’t use the softmax nonlinearity to construct δxy the representations learned by the convolutional PUN are no longer visually interpretable. It is interesting to conclude that under this framework the “what” and “where” can only be distinguished if we impose architectural constraints. The reason is the commutative property of the convolution operation.
As a note on rotation, we ran experiments where the sprite are rotated by a random angle before being placed in the image. This new type of videos cannot be learned using only convolutional based Perception Updating Networks unless we increase the number of sprites proportionally to the number of possible angles. Spatial transformer based Perception Updating Networks can handle this new type of video naturally. Nevertheless, if the number of rotation angles is finite or can be discretized we found that we could learn to generate the videos faster if we combined the convolutional approach with a mechanism to select the appropriate angle from a set of possibilities. Results on this experiment are not shown in this paper due to space constraints but they can be reproduced with the companion code.
4.2 MOVING MNIST
The Moving MNIST benchmark uses videos generated by moving 28×28 pixel images of hand written digits in a 64× 64 pixels canvas. Just like in the Bouncing Shapes dataset, the digits move with different different speeds in different directions and can bounce in the walls. Unlike the Bouncing Shapes dataset, there are 60000 different sprites for training and 10000 for test, making it impractical to use a discrete memory module. Instead, we use the memory representation denoted by (13) followed by st = d(zt) as written in Algorithm 1.
We trained a convolutional Perception Updating Network using 2 layer LSTMs each one with 1024 cells for 200 epochs, with 10000 gradient updates per epoch. The latent variable z had 100 dimensions and the decoder d(·) was a single hidden layer MLP with 1000 hidden neurons and softplus
activation function. The output layer of this MLP has 784 neurons, which is the size of an MNIST image, and sigmoid activation function. In the test set we obtained a negative log-likelihood of 239 nats with the proposed architecture, while a 2 layer LSTM baseline had 250 nats. Note that the our method was optimized to minimize the lower bound (12), not only the negative likelihood. These results are not as good as those obtained by the Video Pixel Networks (Kalchbrenner et al., 2016) that obtained 87 nats on the test set. Nevertheless, both approaches are not mutually exclusive and instead of a fully connected decoder we could use a similar PixelCNN decoder to generate sprites with higher likelihood. In this first paper we decided instead to focus in defining the statistical framework and interpretable “what” and “where” decoupling.
When running the proposed method in rollout mode, feeding the outputs back as next time step inputs, we were able to generate high likelihood frames for more time steps than with a baseline LSTM. Also, since the sprite to be generated and its position in the frame are decoupled, in rollout mode we can fix the sprite and only use the δxy coming from the network. This way we can generate realistic looking frames for even longer, but after a few frames we observed the digits stopped moving or moved in the wrong direction (see video in the companion code repository). This means that the LSTM RNN was not able to maintain its internal dynamics for too long, thus, there is still room for improvement in the proposed architecture.
In Fig. 5 we show sample rollout videos. The network was fed with 10 frames and asked to generate 10 more getting its own outputs back as inputs and the companion code repository for an animated version of this figure.
This experiment also suggests several improvements in the proposed architecture. For example, we assumed that the internal RNN has to calculate a sprite at every time step, which is inefficient when the sprites don’t change in the video. We should improve the architecture with an extra memory unity that snapshots the sprites and avoid the burden of recalculating the sprites at every step. We believe this would a possible way to free representation power that the internal RNN could use to model the movement dynamics for even more time steps.
5 CONCLUSIONS
This paper introduced a statistical framework for modeling video of 2D scenes inspired by graphics pipelines and variational auto-encoding Bayes. From this statistical framework we derived a variational lower bound that decouples sprites and their dynamics in a video. To optimize this lower bound, we suggested a family of architectures called Perception Updating Networks that can take advantage of this decoupled representation by memorizing sprites or their percepts and updating in location in a scene independently. We showed that this architecture could generate videos that are interpretable and are better suited than baseline RNNs for long video generation.
ACKNOWLEDGMENTS
We thank Ryan Burt for several suggestions to the first draft. This work was partially funded by the University of Florida Graduate Student Fellowship and ONR N00014-14-1-0542.

This paper presents a generative model of video sequence data where the frames are assumed to be generated by a static background with a 2d sprite composited onto it at each timestep.  The sprite itself is allowed to dynamically change its appearance and location within the image from frame to frame.  This paper follows the VAE (Variational Autoencoder) approach, where a recognition/inference network allows them to recover the latent state at each timestep.
Some results are presented on simple synthetic data (such as a moving rectangle on a black background or the “Moving MNIST” data.  However, the results are preliminary and I suspect that the assumptions used in the paper are far too strong too be useful in real videos.  On the Moving MNIST data, the numerical results are not competitive to state of the art numbers.
The model itself is also not particularly novel and the work currently misses some relevant citations.  The form of the forward model, for example, could be viewed as a variation on the DRAW paper by Gregor et al (ICML 2014).  Efficient Inference in Occlusion-Aware Generative Models of Images by Huang & Murphy (ICLR) is another relevant work, which used a variational auto-encoder with a spatial transformer and an RNN-like sequence model to model the appearance of multiple sprites on a background.
Finally, the exposition in this paper is short on many details and I don’t believe that the paper is reproducible from the text alone.  For example, it is not clear what the form of the recognition model is…  Low-level details (which are very important) are also not presented, such as initialization strategy.


TIONS WITH NEURAL SIMILARITY AND CONTEXT EN- CODERS
We introduce similarity encoders (SimEc), which learn similarity preserving representations by using a feed-forward neural network to map data into an embedding space where the original similarities can be approximated linearly. The model can easily compute representations for novel (out-of-sample) data points, even if the original pairwise similarities of the training set were generated by an unknown process such as human ratings. This is demonstrated by creating embeddings of both image and text data. Furthermore, the idea behind similarity encoders gives an intuitive explanation of the optimization strategy used by the continuous bag-of-words (CBOW) word2vec model trained with negative sampling. Based on this insight, we define context encoders (ConEc), which can improve the word embeddings created with word2vec by using the local context of words to create out-of-vocabulary embeddings and representations for words with multiple meanings. The benefit of this is illustrated by using these word embeddings as features in the CoNLL 2003 named entity recognition task.
1 INTRODUCTION
Many dimensionality reduction or manifold learning algorithms optimize for retaining the pairwise similarities, distances, or local neighborhoods of data points. Classical scaling (Cox & Cox, 2000), kernel PCA (Schölkopf et al., 1998), isomap (Tenenbaum et al., 2000), and LLE (Roweis & Saul, 2000) achieve this by performing an eigendecomposition of some similarity matrix to obtain a low dimensional representation of the original data. However, this is computationally expensive if a lot of training examples are available. Additionally, out-of-sample representations can only be created when the similarities to the original training examples can be computed (Bengio et al., 2004).
For some methods such as t-SNE (van der Maaten & Hinton, 2008), great effort was put into extending the algorithm to work with large datasets (van der Maaten, 2013) or to provide an explicit mapping function which can be applied to new data points (van der Maaten, 2009). Current attempts at finding a more general solution to these issues are complex and require the development of specific cost functions and constraints when used in place of existing algorithms (Bunte et al., 2012), which limits their applicability to new objectives.
In this paper we introduce a new neural network architecture, that we will denote as similarity encoder (SimEc), which is able to learn representations that can retain arbitrary pairwise relations present in the input space, even those obtained from unknown similarity functions such as human ratings. A SimEc can learn a linear or non-linear mapping function to project new data points into a lower dimensional embedding space. Furthermore, it can take advantage of large datasets since the objective function is optimized iteratively using stochastic mini-batch gradient descent. We show on both image and text datasets that SimEcs can, on the one hand, recreate solutions found by traditional methods such as kPCA or isomap, and, on the other hand, obtain meaningful embeddings from similarities based on human labels.
Additionally, we propose the new context encoder (ConEc) model, a variation of similarity encoders for learning word embeddings, which extends word2vec (Mikolov et al., 2013b) by using the local context of words as input to the neural network to create representations for out-of-vocabulary words and to distinguish between multiple meanings of words. This is shown to be advantageous, for example, if the word embeddings are used as features in a named entity recognition task as demonstrated on the CoNLL 2003 challenge.
2 SIMILARITY ENCODERS
We propose a novel dimensionality reduction framework termed similarity encoder (SimEc), which can be used to learn a linear or non-linear mapping function for computing low dimensional representations of data points such that the original pairwise similarities between the data points in the input space are preserved in the embedding space. For this, we borrow the “bottleneck” neural network (NN) architecture idea from autoencoders (Tishby et al., 2000; Hinton & Salakhutdinov, 2006). Autoencoders aim to transform the high dimensional data points into low dimensional embeddings such that most of the data’s variance is retained. Their network architecture has two parts: The first part of the network maps the data points from the original feature space to the low dimensional embedding (at the bottleneck). The second part of the NN mirrors the first part and projects the embedding back to a high dimensional output. This output is then compared to the original input to compute the reconstruction error of the training samples, which is used in the backpropagation procedure to tune the network’s parameters. After the training is complete, i.e. the low dimensional embeddings encode enough information about the original input samples to allow for their reconstruction, the second part of the network is discarded and only the first part is used to project data points into the low dimensional embedding space. Similarity encoders have a similar two fold architecture, where in the first part of the network, the data is mapped to a low dimensional embedding, and then in the second part (which is again only used during training), the embedding is transformed such that the error of the representation can be computed. However, since here the objective is to retain the (non-linear) pairwise similarities instead of the data’s variance, the second part of the NN does not mirror the first like it does in the autoencoder architecture.
The similarity encoder architecture (Figure 1) uses as the first part of the network a flexible non-linear feed-forward neural network to map the high dimensional input data points xi ∈ RD to a low dimensional embedding yi ∈ Rd (at the bottleneck). As we make no assumptions on the range of values the embedding can take, the last layer of the first part of the NN (i.e. the one resulting in the embedding) is always linear. For example, with two additional non-linear hidden layers, the embedding would be computed as
yi = σ1(σ0(xiW0)W1)W2,
where σ0 and σ1 denote your choice of non-linear activation functions (e.g. tanh, sigmoid, or relu), but there is no non-linearity applied after multiplying with W2. The second part of the network then
consists of a single additional layer with the weight matrix W−1 ∈ Rd×N to project the embedding to the output, the approximated similarities s′ ∈ RN :
s′ = σ−1(yiW−1).
These approximated similarities are then compared to the target similarities (for one data point this is the corresponding row si ∈ RN of the similarity matrix S ∈ RN×N of the N training samples) and the computed error is used to tune the network’s parameters with backpropagation.
For the model to learn most efficiently, the exact form of the cost function to optimize as well as the type of non-linearity σ−1 applied when computing the network’s output should be chosen with respect to the type of target similarities that the model is supposed to preserve. In the experimental section of the paper we are considering two application scenarios of SimEcs: a) to obtain the same low dimensional embedding as found by spectral methods such as kPCA, and b) to embed data points such that binary similarity relations obtained from human labels are preserved. In the first case (further discussed in the next section), we omit the non-linearity when computing the output of the network, i.e. s′ = yiW−1, since the target similarities, computed by some kernel function, are not necessarily constrained to lie in a specific interval. As the cost function to minimize we choose the mean squared error between the output (approximated similarities) and the original (target) similarities. A regularization term is added to encourage the weights of the last layer (W−1) to be orthogonal.1 The model’s objective function optimized during training is therefore:
min 1
N N∑ i=1 ‖si − s′‖22 + λ 1 d2 − d ∥∥W−1W>−1 − diag(W−1W>−1)∥∥1
where ‖ · ‖p denotes the respective p-norms for vectors and matrices and λ is a hyperparameter to control the strength of the regularization. In the second case, the target similarities are binary and it therefore makes sense to use a nonlinear activation function in the final layer when computing the output of the network to ensure the approximated similarities are between 0 and 1 as well:2
s′ = σ−1(yiW−1) with σ−1(z) = 1
1 + e−10(z−0.5) .
While the mean squared error between the target and approximated similarities would still be a natural choice of cost function to optimize, with the additional non-linearity in the output layer, learning might be slow due to small gradients and we therefore instead optimize the cross-entropy:
min − 1 N
∑ [si ln(s ′) + (1− si) ln(1− s′)] .
For a different application scenario, yet another setup might lead to the best results. When using SimEcs in practice, we recommend to first try the first setup, i.e. keeping the output layer linear and minimizing the mean squared error, as this often already gives quite good results.
After the training is completed, only the first part of the neural network, which maps the input to the embedding, is used to create the representations of new data points. Depending on the complexity of the feed-forward NN, the mapping function learned by similarity encoders can be linear or non-linear, and because of the iterative optimization using stochastic mini-batch gradient descent, large amounts of data can be utilized to learn optimal representations.3
2.1 RELATION TO KERNEL PCA
Kernel PCA (kPCA) is a popular non-linear dimensionality reduction algorithm, which performs the eigendecomposition of a kernel matrix to obtain low dimensional representations of the data points
1To get embeddings similar to those obtained by kPCA, orthogonal weights in the last layer of the NN help as they correspond to the orthogonal eigenvectors of the kernel matrix found by kPCA.
2This scaled and shifted sigmoid function maps values between 0 and 1 almost linearly while thresholding values outside this interval.
3To speed up the training procedure and limit memory requirements for large datasets, the columns of the similarity matrix can also be subsampled (yielding S ∈ RN×n), i.e. the number of target similarities (and the dimensionality of the output layer) is n < N , however all N training examples can still be used as input to train the network.
(Schölkopf et al., 1998). However, if the kernel matrix is very large this becomes computationally very expensive. Additionally, there are constraints on possible kernel functions (should be positive semi-definite) and new data points can only be embedded in the lower dimensional space if their kernel map (i.e. the similarities to the original training points) can be computed. As we show below, SimEc can optimize the same objective as kPCA but addresses these shortcomings.
The general idea is that both kPCA and SimEc embed the N data points in a feature space where the given target similarities can be approximated linearly (i.e. with the scalar product of the embedding vectors). When the error between the approximated (S′) and the target similarities (S) is computed as the mean squared error, kPCA finds the optimal approximation by performing the eigendecomposition of the (centered) target similarity matrix, i.e.
S′ = Y Y >,
where Y ∈ RN×d is the low dimensional embedding of the data based on the eigenvectors belonging to the d largest eigenvalues of S.
In addition to the embedding itself, it is often desired to have a parametrized mapping function, which can be used to project new (out-of-sample) data points into the embedding space. If the target similarity matrix is the linear kernel, i.e. S = XX> where X ∈ RN×D is the given input data, this can easily be accomplished with traditional PCA. Here, the covariance matrix of the centered input data, i.e. C = X>X is decomposed to obtain a matrix with parameters, W̃ ∈ RD×d, based on the eigenvectors belonging to the d largest eigenvalues of the covariance matrix. Then the optimal embedding (i.e. the same solution obtained by linear kPCA) can be computed as
Y = XW̃.
This serves as a mapping function, with which new data points can be easily projected into the lower dimensional embedding space.
When using a similarity encoder to embed data in a low dimensional space where the linear similarities are preserved, the SimEc’s architecture would consist of a neural network with a single linear layer, i.e. the parameter matrix W0, to project the input data X to the embedding Y = XW0, and another matrix W−1 ∈ Rd×N used to approximate the similarities as S′ = YW−1. From these formulas one can immediately see the link between linear similarity encoders and PCA / linear kPCA: once the parameters of the neural network are tuned correctly, W0 would correspond to the mapping matrix W̃ found by PCA and W−1 could be interpreted as Y >, i.e. Y would be the same eigenvector based embedding as found with linear kPCA.
Finding the corresponding function to map new data points into the embedding space is trivial for linear kPCA, but this is not the case for other kernel functions. While it is still possible to find the optimal embedding with kPCA for non-linear kernel functions, the mapping function remains unknown and new data points can only be projected into the embedding space if we can compute their kernel map, i.e. the similarities to the original training examples (Bengio et al., 2004). Some attempts were made to manually define an explicit mapping function to represent data points in the kernel feature space, however this only works for specific kernels and there exists no general solution (Rahimi & Recht, 2007). As neural networks are universal function approximators, with the right architecture similarity encoders could instead learn arbitrary mapping functions for unknown similarities to arrive at data driven kernel learning solutions.
2.2 MODEL OVERVIEW
The properties of similarity encoders are summarized in the following. The objective of this dimensionality reduction approach is to retain pairwise similarities between data points in the embedding space. This is achieved by tuning the parameters of a neural network to obtain a linear or non-linear mapping (depending on the network’s architecture) from the high dimensional input to the low dimensional embedding. Since the cost function is optimized using stochastic mini-batch gradient descent, we can take advantage of large datasets for training. The embedding for new test points can be easily computed with the explicit mapping function in the form of the tuned neural network. And since there is no need to compute the similarity of new test examples to the original training data for out-of-sample solutions (like with kPCA), the target similarities can be generated by an unknown process such as human similarity judgments.
2.3 EXPERIMENTS
In the following experiments we demonstrate that similarity encoders can, on the one hand, reach the same solution as kPCA, and, on the other hand, generate meaningful embeddings from human labels. To illustrate that this is independent of the type of data, we present results obtained both on the well known MNIST handwritten digits dataset as well as the 20 newsgroups text corpus. Further details as well as the code to replicate these experiments and more is available online.4
We compare the embedding found with linear kPCA to that created with a linear similarity encoder (consisting of one linear layer mapping the input to the embedding and a second linear layer to project the embedding to the output, i.e. computing the approximated similarities). Additionally, we show that a non-linear SimEc can approximate the solution found with isomap (i.e. the eigendecomposition of the geodesic distance matrix). We found that for optimal results the kernel matrix used as the target similarity matrix for the SimEc should first be centered (as it is being done for kPCA as well (Müller et al., 2001)).
In a second step, we show that SimEcs can learn the mapping to a low dimensional embedding for arbitrary similarity functions and reliably create representations for new test samples without the need to compute their similarities to the original training examples, thereby going beyond the capabilities of kPCA. For both datasets we illustrate this by using the class labels assigned to the samples by human annotators to create the target similarity matrix for the training fold of the data, i.e. S is 1 for data points belonging to the same class and 0 everywhere else. We compare the solutions found by SimEc architectures with a varying number of additional non-linear hidden layers in the first part of the network (while keeping the embedding layer linear as before) to show how a more complex network improves the ability to map the data into an embedding space in which the class-based similarities are retained.
MNIST The MNIST dataset contains 28× 28 pixel images depicting handwritten digits. For our experiments we randomly subsampled 10k images from all classes, of which 80% are assigned to the training fold and the remaining 20% to the test fold (in the following plots, data points belonging to the training set are displayed transparently while the test points are opaque). As shown in Figure 2, the embeddings of the MNIST dataset created with linear kPCA and a linear similarity encoder, which uses as target similarities the linear kernel matrix, are almost identical (up to a rotation). The same holds true for the isomap embedding, which is well approximated by a non-linear SimEc with two hidden layers using the geodesic distances between the data points as targets (Figure 8 in the Appendix). When optimizing SimEcs to retain the class-based similarities (Figure 3), additional
non-linear hidden layers in the feed-forward NN can improve the embedding by further separating data points belonging to different classes in tight clusters. As it can be seen, the test points (opaque) are nicely mapped into the same locations as the corresponding training points (transparent), i.e. the model learns to associate the input pixels with the class clusters only based on the imposed similarities between the training data points.
4https://github.com/cod3licious/simec/examples_simec.ipynb
20 newsgroups The 20 newsgroups dataset consists of around 18k newsgroup posts assigned to 20 different topics. We take a subset of seven categories and use the original train/test split (∼4.1k and ∼2.7k samples respectively) and remove metadata such as headers to avoid overfitting.5 All text documents are transformed into 46k dimensional tf-idf feature vectors, which are used as input to the SimEc and to compute the linear kernel matrix of the training fold. The embedding created with linear kPCA is again well approximated by the solution found with a corresponding linear SimEc (Figure 9 in the Appendix). Additionally, this serves as an example where traditional PCA is not an option to obtain the corresponding mapping matrix for the linear kPCA solution, as due to the high dimensionality of the input data and comparatively low number of samples, the empirical covariance matrix would be poorly estimated and too large to decompose into eigenvalues and -vectors. With the objective to retain the class-based similarities, a SimEc with a non-linear hidden layer clusters documents by their topics (Figure 4).
3 CONTEXT ENCODERS
Representation learning is very prominent in the field of natural language processing (NLP). For example, word embeddings learned by neural network language models were shown to improve the performance when used as features for supervised learning tasks such as named entity recognition (NER) (Collobert et al., 2011; Turian et al., 2010). The popular word2vec model (Figure 5) learns meaningful word embeddings by considering only the words’ local contexts and thanks to its shallow architecture it can be trained very efficiently on large corpora. However, an important limiting factor of current word embedding models is that they only learn the representations for words from a fixed vocabulary. This means, if in a task we encounter a new word which was not present in the texts used for training, we can not create an embedding for this word without repeating the time consuming
5http://scikit-learn.org/stable/datasets/twenty_newsgroups.html
training procedure of the model.6 Additionally, word2vec, like many other approaches, only learns a single representation for every word. However, it is often the case that a single word can have multiple meanings, e.g. “Washington” is both the name of a US state as well as a former president. It is only the local context in which these words appear that lets humans resolve this ambiguity and identify the proper sense of the word in question. While attempts were made to improve this, they lack flexibility as they require a clustering of word contexts beforehand (Huang et al., 2012), which still does not guarantee that all possible meanings of a word have been identified prior in the training documents. Other approaches require additional labels such part-of-speech tags (Trask et al., 2015) or other lexical resources like WordNet (Rothe & Schütze, 2015) to create word embeddings which distinguish between the different senses of a word.
As a further contribution of this paper we provide a link between the successful word2vec natural language model and similarity encoders and thereby propose a new model we call context encoder (ConEc), which can efficiently learn word embeddings from huge amounts of training data and additionally make use of local contexts to create representations for out-of-vocabulary words and help distinguish between multiple meanings of words.
6In practice these models are trained on such a large vocabulary that it is rare to encounter a word which does not have an embedding. However, there are still scenarios where this is the case, for example, it is unlikely that the term “W10281545” is encountered in a regular training corpus, but we might still want its embedding to represent a search query like “whirlpool W10281545 ice maker part”.
Formally, word embeddings are d -dimensional vector representations learned for all N words in the vocabulary. Word2vec is a shallow model with parameter matrices W0,W1 ∈ RN×d, which are tuned iteratively by scanning huge amounts of texts sentence by sentence (see Figure 5). Based on some context words the algorithm tries to predict the target word between them. Mathematically this is realized by first computing the sum of the embeddings of the context words by selecting the appropriate rows from W0. This vector is then multiplied by several rows selected from W1: one of these rows corresponds to the target word, while the others correspond to k ‘noise’ words, selected at random (negative sampling). After applying a non-linear activation function, the backpropagation error is computed by comparing this output to a label vector t ∈ Rk+1, which is 1 at the position of the target word and 0 for all k noise words. After the training of the model is complete, the word embedding for a target word is the corresponding row of W0.
The main principle utilized when learning word embeddings is that similar words appear in similar contexts (Harris, 1954; Melamud et al., 2015). Therefore, in theory one could compute the similarities between all words by checking how many context words any two words generally have in common (possibly weighted somehow to reduce the influence of frequent words such as ‘the’ and ‘and’). However, such a word similarity matrix would be very large, as typically the vocabulary for which word embeddings are learned comprises several 10, 000 words, making it computationally too expensive to be used with similarity encoders. But this matrix would also be quite sparse, because many words in fact do not occur in similar contexts and most words only have a handful of synonyms which could be used in their place. Therefore, we can view the negative sampling approach used for word2vec (Mikolov et al., 2013b) as an approximation of the words’ context based similarities: while the similarity of a word to itself is 1, if for one word we select k random words out of the huge vocabulary, it is very unlikely that they are similar to the target word, i.e. we can approximate their similarities with 0. This is the main insight necessary for adapting similarity encoders to be used for learning (context sensitive) word embeddings.
Figure 6 shows the architecture of the context encoder. For the training procedure we stick very closely to the optimization strategy used by word2vec: while parsing a document, we again select a target word and its context words. As input to the context encoder network, we use a vector xi of length N (i.e. the size of the vocabulary), which indicates the context words by non-zero values (either binary or e.g. giving lower weight to context words further away from the target word). This vector is then multiplied by a first matrix of weights W0 ∈ RN×d yielding a low dimensional embedding yi, comparable to the summed context embedding created as a first step when training the word2vec model. This embedding is then multiplied by a second matrix W1 ∈ Rd×N to yield the output. Instead of comparing this output vector to a whole row from a word similarity matrix (as we would with similarity encoders), only k + 1 entries are selected, namely those belonging to
the target word as well as k random and unrelated noise words. After applying a non-linearity we compare these entries s′ ∈ Rk+1 to the binary target vector exactly as in the word2vec model and use error backpropagation to tune the parameters.
Up to now, there are no real differences between the word2vec model and our context encoders, we have merely provided an intuitive interpretation of the training procedure and objective. The main deviation from the word2vec model lies in the computation of the word embedding for a target word after the training is complete. In the case of word2vec, the word embedding is simply the row of the tuned W0 matrix. However, when considering the idea behind the optimization procedure, we instead propose to compute a target word’s representation by multiplying W0 with the word’s average context vector. This is closer to what is being done in the training procedure and additionally it enables us to compute the embeddings for out-of-vocabulary words (assuming at least most of such a new word’s context words are in the vocabulary) as well as to place more emphasis on a word’s local context (which helps to identify the proper meaning of the word (Melamud et al., 2015)) by creating a weighted sum between the word’s average global and local context vectors used as input to the ConEc.
With this new perspective on the model and optimization procedure, another advancement is feasible. Since the context words are merely a sparse feature vector used as input to a neural network, there is no reason why this input vector should not contain other features about the target word as well. For example, the feature vector could be extended to contain information about the word’s case, part-of-speech (POS) tag, or other relevant details. While this would increase the dimensionality of the first weight matrix W0 to include the additional features when mapping the input to the word’s embedding, the training objective and therefore also W1 would remain unchanged. These additional features could be especially helpful if details about the words would otherwise get lost in preprocessing (e.g. by lowercasing) or to retain information about a word’s position in the sentence, which is ignored in a BOW approach. These extended ConEcs are expected to create embeddings which distinguish even better between the words’ different senses by taking into account, for example, if the word is used as a noun or verb in the current context, similar to the sense2vec algorithm (Trask et al., 2015). However, unlike sense2vec, not multiple embeddings per term are learned, instead the dimensionality of the input vector is increased to include the POS tag of the current word as a feature.
3.1 EXPERIMENTS
The word embeddings learned with word2vec and context encoders are evaluated on a word analogy task (Mikolov et al., 2013a) as well as the CoNLL 2003 NER benchmark task (Tjong et al., 2003). The word2vec model used is a continuous BOW model trained with negative sampling as described above where k = 13, the embedding dimensionality d is 200 and we use a context window of 5. The word embeddings created by the context encoders are build directly on top of the word2vec model by multiplying the original embeddings (W0) with the respective context vectors. Code to replicate the experiments can be found online.7 The results of the analogy task can be found in the Appendix.8
Named Entity Recognition The main advantage of context encoders is that they can use local context to create out-of-vocabulary (OOV) embeddings and distinguish between the different senses of words. The effects of this are most prominent in a task such as named entity recognition (NER) where the local context of a word can make all the difference, e.g. to distinguish between the “Chicago Bears” (an organization) and the city of Chicago (a location). To test this, we used the word embeddings as features in the CoNLL 2003 NER benchmark task (Tjong et al., 2003). The word2vec embeddings were trained on the documents used in the training part of the task.9 For the context encoders we experimented with different combinations of local and global context vectors. The global context vectors were computed on only the training documents as well, i.e. just as with
7https://github.com/cod3licious/conec 8As it was recently demonstrated that a good performance on intrinsic evaluation tasks such as word similarity or analogy tasks does not necessarily transfer to extrinsic evaluation measures when using the word embeddings as features (Chiu et al., 2016; Linzen, 2016), we consider the performance on the NER challenge as more relevant.
9Since this is a very small corpus, we trained word2vec for 25 iterations on these documents (afterwards the performance on the development split stopped improving significantly) while usually the model is trained in a single pass through a much larger corpus.
the word2vec model, when applied to the test documents there are some words which don’t have a word embedding available as they did not occur in the training texts. The local context vectors on the other hand can be computed for all words occurring in the current document for which the model should identify the named entities. When combining these local context vectors with the global ones we always use the local context vector as is in case there is no global vector available and otherwise compute a weighted average between the two context vectors as wl · CVlocal + (1− wl) · CVglobal.10 The different word embeddings were used as features with a logistic regression classifier trained on the labels obtained from the training part of the task and the reported F1-scores were computed using the official evaluation script. Please note that we are using this task to show the potential of ConEc word embeddings as features in a real world task and to illustrate their advantages over the regular word2vec embeddings and did not optimize for competitive performance on this NER challenge.
Figure 7 shows the results achieved with various word embeddings on the training, development and test part of the CoNLL task. As it can be seen there, taking into account the local context can yield large improvements, especially on the dev and test data. Context encoders using only the global context vectors already perform better than word2vec. When using the local context vectors only where the global ones are not available (wl = 0) we can see a jump in the development and test performance, while of course the training performance stays the same as here we have global context vectors for all words. The best performances on all folds are achieved when averaging the global and local context vectors with around wl = 0.4 before multiplying them with the word2vec embeddings. This clearly shows that using ConEcs with local context vectors can be very beneficial as they let us compute word embeddings for out-of-vocabulary words as well as help distinguish between multiple meanings of words.
10The global context matrix is computed without taking the word itself into account (i.e. zero on the diagonal) to make the context vectors comparable to the local context vectors of OOV words where we can’t count the target word either. Both global and local context vectors are normalized by their respective maximum values, then multiplied with the length normalized word2vec embeddings and again renormalized to have unit length.
4 CONCLUSION
Representing intrinsically complex data is an ubiquitous challenge in data analysis. While kernel methods and manifold learning have made very successful contributions, their ability to scale is somewhat limited. Neural autoencoders offer scalable nonlinear embeddings, but their objective is to minimize the reconstruction error of the input data which does not necessarily preserve important pairwise relations between data points. In this paper we have proposed SimEcs as a neural network framework which bridges this gap by optimizing the same objective as spectral methods, such as kPCA, for creating similarity preserving embeddings while retaining the favorable properties of autoencoders.
Similarity encoders are a novel method to learn similarity preserving embeddings and can be especially useful when it is computationally infeasible to perform the eigendecomposition of a kernel matrix, when the target similarities are obtained through an unknown process such as human similarity judgments, or when an explicit mapping function is required. To accomplish this, a feed-forward neural network is constructed to map the data into an embedding space where the original similarities can be approximated linearly.
As a second contribution we have defined context encoders, a practical extension of SimEcs, that can be readily used to enhance the word2vec model with further local context information and global word statistics. Most importantly, ConEcs allow to easily create word embeddings for out-of-vocabulary words on the spot and distinguish between different meanings of a word based its local context.
Finally, we have demonstrated the usefulness of SimEcs and ConEcs for practical tasks such as the visualization of data from different domains and to create meaningful word embedding features for a NER task, going beyond the capabilities of traditional methods.
Future work will aim to further the theoretical understanding of SimEcs and ConEcs and explore other application scenarios where using this novel neural network architecture can be beneficial. As it is often the case with neural network models, determining the optimal architecture as well as other hyperparameter choices best suited for the task at hand can be difficult. While so far we mainly studied SimEcs based on fairly simple feed-forward networks, it appears promising to consider also deeper neural networks and possibly even more elaborate architectures, such as convolutional networks, for the initial mapping step to the embedding space, as in this manner hierarchical structures in complex data could be reflected. Note furthermore that prior knowledge as well as more general error functions could be employed to tailor the embedding to the desired application target(s).
ACKNOWLEDGMENTS
We would like to thank Antje Relitz, Christoph Hartmann, Ivana Balažević, and other anonymous reviewers for their helpful comments on earlier versions of this manuscript. Additionally, Franziska Horn acknowledges funding from the Elsa-Neumann scholarship from the TU Berlin.

This paper introduces a similarity encoder based on a standard feed-forward neural network with the aim of generating similarity-preserving embeddings. The approach is utilized to generate a simple extension of the CBOW word2vec model that transforms the learned embeddings by their average context vectors. Experiments are performed on an analogy task and named entity recognition.
While this paper offers some reasonable intuitive arguments for why a feed-forward neural network can generate good similarity-preserving embeddings, the architecture and approach is far from novel. As far as I can tell, the model is nothing more than the most vanilla neural network trained with SGD on similarity signals.
Slightly more original is the idea to use context embeddings to augment the expressive capacity of learned word representations. Of course, using explicit contextual information is not a new idea, especially for tasks like word sense disambiguation (see, e.g., 'Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space' by Neelakantan et al, which should also be cited), but the specific method used here is original, as far as I know.
The evaluation of the method is far from convincing. The corpora used to train the embeddings are far smaller than would ever be used in practice for unsupervised or semi-supervised embedding learning. The performance on the analogy task says little about the benefit of this method for larger corpora, and, as the authors mentioned in the comments, they expect "the gain will be less significant, as the global context statistics brought in by the ConEc can also be picked up by word2vec with more training."
The argument can be made (and the authors do claim) that extrinsic evaluations are more important for real-world applications, so it is good to see experiments on NER. However, again the embeddings were trained on a very small corpus and I am not convinced that the observed benefit will persist when trained on larger corpora.
Overall, I believe this paper offers little novelty and weak experimental evidence supporting its claims. I cannot recommend it for acceptance.