Attention plays a critical role in human visual experience. Furthermore, it has recently been demonstrated that attention can also play an important role in the context of applying artificial neural networks to a variety of tasks from fields such as computer vision and NLP. In this work we show that, by properly defining attention for convolutional neural networks, we can actually use this type of information in order to significantly improve the performance of a student CNN network by forcing it to mimic the attention maps of a powerful teacher network. To that end, we propose several novel methods of transferring attention, showing consistent improvement across a variety of datasets and convolutional neural network architectures. Code and models for our experiments are available at https://github.com/szagoruyko/attention-transfer.
1 INTRODUCTION
As humans, we need to pay attention in order to be able to adequately perceive our surroundings. Attention is therefore a key aspect of our visual experience, and closely relates to perception - we need to keep attention to build a visual representation, possessing detail and coherence.
As artificial neural networks became more popular in fields such as computer vision and natural language processing in the recent years, artificial attention mechanisms started to be developed as well. Artificial attention lets a system “attend” to an object to examine it with greater detail. It has also become a research tool for understanding mechanisms behind neural networks, similar to attention used in psychology.
One of the popular hypothesis there is that there are non-attentional and attentional perception processes. Non-attentional processes help to observe a scene in general and gather high-level information, which, when associated with other thinking processes, helps us to control the attention processes and navigate to a certain part of the scene. This implies that different observers with different knowledge, different goals, and therefore different attentional strategies can literally see the same scene differently. This brings us to the main topic of this paper: how attention differs within artificial vision systems, and can we use attention information in order to improve the performance of convolutional neural networks ? More specifically, can a teacher network improve the performance of another student network by providing to it information about where it looks, i.e., about where it concentrates its attention into ?
To study these questions, one first needs to properly specify how attention is defined w.r.t. a given convolutional neural network. To that end, here we consider attention as a set of spatial maps that essentially try to encode on which spatial areas of the input the network focuses most for taking its output decision (e.g., for classifying an image), where, furthermore, these maps can be defined w.r.t. various layers of the network so that they are able to capture both low-, mid-, and high-level representation information. More specifically, in this work we define two types of spatial attention maps: activation-based and gradient-based. We explore how both of these attention maps change over various datasets and architectures, and show that these actually contain valuable information
that can be used for significantly improving the performance of convolutional neural network architectures (of various types and trained for various different tasks). To that end, we propose several novel ways of transferring attention from a powerful teacher network to a smaller student network with the goal of improving the performance of the latter (Fig. 1).
To summarize, the contributions of this work are as follows:
• We propose attention as a mechanism of transferring knowledge from one network to another
• We propose the use of both activation-based and gradient-based spatial attention maps • We show experimentally that our approach provides significant improvements across a va-
riety of datasets and deep network architectures, including both residual and non-residual networks
• We show that activation-based attention transfer gives better improvements than fullactivation transfer, and can be combined with knowledge distillation
The rest of the paper is structured as follows: we first describe related work in section 2, we explain our approach for activation-based and gradient-based attention transfer in section 3, and then present experimental results for both methods in section 4. We conclude the paper in section 5.
2 RELATED WORK
Early work on attention based tracking Larochelle & Hinton (2010), Denil et al. (2012) was motivated by human attention mechanism theories Rensink (2000) and was done via Restricted Bolzmann Machines. It was recently adapted for neural machine translation with recurrent neural networks, e.g. Bahdanau et al. (2014) as well as in several other NLP-related tasks. It was also exploited in computer-vision-related tasks such as image captioning Xu et al. (2015), visual question answering Yang et al. (2015), as well as in weakly-supervised object localization Oquab et al. (2015) and classification Mnih et al. (2014), to mention a few characteristic examples. In all these tasks attention proved to be useful.
Visualizing attention maps in deep convolutional neural networks is an open problem. The simplest gradient-based way of doing that is by computing a Jacobian of network output w.r.t. input (this leads to attention visualization that are not necessarily class-discriminative), as for example in Simonyan et al. (2014). Another approach was proposed by Zeiler & Fergus (2014) that consists of attaching a network called “deconvnet” that shares weights with the original network and is used to project certain features onto the image plane. A number of methods was proposed to improve gradientbased attention as well, for example guided backpropagation Springenberg et al. (2015), adding a change in ReLU layers during calculation of gradient w.r.t. previous layer output. Attention maps obtained with guided backpropagation are non-class-discriminative too. Among existing methods
for visualizing attention, we should also mention class activation maps Zhou et al. (2016), which are based on removing top average-pooling layer and converting the linear classification layer into a convolutional layer, producing attention maps per each class. A method combining both guided backpropagation and CAM is Grad-CAM by Selvaraju et al. (2016), adding image-level details to class-discriminative attention maps.
Knowledge distillation with neural networks was pioneered by Hinton et al. (2015); Bucila et al. (2006), which is a transfer learning method that aims to improve the training of a student network by relying on knowledge borrowed from a powerful teacher network. Although in certain special cases shallow networks had been shown to be able to approximate deeper ones without loss in accuracy Lei & Caruana (2014), later work related to knowledge distillation was mostly based on the assumption that deeper networks always learn better representations. For example, FitNets Romero et al. (2014) tried to learn a thin deep network using a shallow one with more parameters. The introduction of highway Srivastava et al. (2015) and later residual networks He et al. (2015) allowed training very deep architectures with higher accuracy, and generality of these networks was experimentally showed over a large variety of datasets. Although the main motivation for residual networks was increasing depth, it was later shown by Zagoruyko & Komodakis (2016) that, after a certain depth, the improvements came mostly from increased capacity of the networks, i.e. number of parameters (for instance, a wider deep residual network with only 16 layers was shown that it could learn as good or better representations as very thin 1000 layer one, provided that they were using comparable number of parameters).
Due to the above fact and due to that thin deep networks are less parallelizable than wider ones, we think that knowledge transfer needs to be revisited, and take an opposite to FitNets approach - we try to learn less deep student networks. Our attention maps used for transfer are similar to both gradient-based and activation-based maps mentioned above, which play a role similar to “hints” in FitNets, although we don’t introduce new weights.
3 ATTENTION TRANSFER
In this section we explain the two methods that we use for defining the spatial attention maps of a convolutional neural network as well as how we transfer attention information from a teacher to a student network in each case.
3.1 ACTIVATION-BASED ATTENTION TRANSFER
Let us consider a CNN layer and its corresponding activation tensor A ∈ RC×H×W , which consists of C feature planes with spatial dimensionsH×W . An activation-based mapping function F (w.r.t. that layer) takes as input the above 3D tensor A and outputs a spatial attention map, i.e., a flattened 2D tensor defined over the spatial dimensions, or
F : RC×H×W → RH×W . (1)
To define such a spatial attention mapping function, the implicit assumption that we make in this section is that the absolute value of a hidden neuron activation (that results when the network is evaluated on given input) can be used as an indication about the importance of that neuron w.r.t. the specific input. By considering, therefore, the absolute values of the elements of tensor A, we can construct a spatial attention map by computing statistics of these values across the channel dimension (see Fig. 3). More specifically, in this work we will consider the following activation-based spatial attention maps:
• sum of absolute values: Fsum(A) = ∑C i=1 |Ai|
• sum of absolute values raised to the power of p (where p > 1): F psum(A) = ∑C i=1 |Ai|p
• max of absolute values raised to the power of p (where p > 1): F pmax(A) = maxi=1,C |Ai|p
where Ai = A(i, :, :) (using Matlab notation), and max, power and absolute value operations are elementwise (e.g. |Ai|p is equivalent to abs(Ai).∧p in Matlab notation).
We visualized activations of various networks on several datasets, including ImageNet classification and localization, COCO object detection, face recognition, and fine-grained recognition. We were mostly focused on modern architectures without top dense linear layers, such as Network-InNetwork, ResNet and Inception, which have streamlined convolutional structure. We also examined networks of the same architecture, width and depth, but trained with different frameworks with significant difference in performance. We found that the above statistics of hidden activations not only have spatial correlation with predicted objects on image level, but these correlations also tend to be higher in networks with higher accuracy, and stronger networks have peaks in attention where weak networks don’t (e.g., see Fig. 4). Furthermore, attention maps focus on different parts for different layers in the network. In the first layers neurons activation level is high for low-level gradient points, in the middle it is higher for the most discriminative regions such as eyes or wheels, and in the top layers it reflects full objects. For example, mid-level attention maps of a network trained for face recognition Parkhi et al. (2015) will have higher activations around eyes, nose and lips, and top level activation will correspond to full face (Fig. 2).
Concerning the different attention mapping functions defined above, these can have slightly different properties. E.g.:
• Compared to Fsum(A), the spatial map F psum(A) (where p > 1) puts more weight to spatial locations that correspond to the neurons with the highest activations, i.e., puts more weight to the most discriminative parts (the larger the p the more focus is placed on those parts with highest activations).
• Furthermore, among all neuron activations corresponding to the same spatial location, F pmax(A) will consider only one of them to assign a weight to that spatial location (as opposed to F psum(A) that will favor spatial locations that carry multiple neurons with high activations).
To further illustrate the differences of these functions we visualized attention maps of 3 networks with sufficient difference in classification performance: Network-In-Network (62% top-1 val accuracy), ResNet34 (73% top-1 val accuracy) and ResNet-101 (77.3% top-1 val accuracy). In each network we took last pre-downsampling activation maps, on the left for mid-level and on the right for top pre-average pooling activations in fig. 4. Top-level maps are blurry because their original spatial resolution is 7 × 7. It is clear that most discriminative regions have higher activation levels, e.g. face of the wolf, and that shape details disappear as the parameter p (used as exponent) increases.
In attention transfer, given the spatial attention maps of a teacher network (computed using any of the above attention mapping functions), the goal is to train a student network that will not only make correct predictions but will also have attentions maps that are similar to those of the teacher. In general, one can place transfer losses w.r.t. attention maps computed across several layers. For instance, in the case of ResNet architectures, one can consider the following two cases, depending on the depth of teacher and student:
• Same depth: possible to have attention transfer layer after every residual block
• Different depth: have attention transfer on output activations of each group of residual blocks
Similar cases apply also to other architectures (such as NIN, in which case a group refers to a block of a 3 × 3, 1 × 1, 1 × 1 convolutions). In fig. 5 we provide a schematic illustration of the different depth case for residual network architectures.
Without loss of generality, we assume that transfer losses are placed between student and teacher attention maps of same spatial resolution, but, if needed, attention maps can be interpolated to match their shapes. Let S, T and WS , WT denote student, teacher and their weights correspondingly, and let L(W, x) denote a standard cross entropy loss. Let also I denote the indices of all teacher-student activation layer pairs for which we want to transfer attention maps. Then we can define the following total loss:
LAT = L(WS , x) + β
2 ∑ j∈I ‖ QjS ‖QjS‖2 − QjT ‖QjT ‖2 ‖p , (2)
where QjS = vec(F (A j S)) and Q j T = vec(F (A j T )) are respectively the j-th pair of student and teacher attention maps in vectorized form, and p refers to norm type (in the experiments we use p = 2). As can be seen, during attention transfer we make use of l2-normalized attention maps, i.e., we replace each vectorized attention map Q with Q‖Q‖2 (l1 normalization could be used as well). It
is worth emphasizing that normalization of attention maps is important for the success of the student training.
Attention transfer can also be combined with knowledge distillation Hinton et al. (2015), in which case an additional term (corresponding to the cross entropy between softened distributions over labels of teacher and student) simply needs to be included to the above loss. When combined, attention transfer adds very little computational cost, as attention maps for teacher can be easily computed during forward propagation, needed for distillation.
3.2 GRADIENT-BASED ATTENTION TRANSFER
In this case we define attention as gradient w.r.t. input, which can be viewed as an input sensitivity map Simonyan et al. (2014), i.e., attention at an input spatial location encodes how sensitive the output prediction is w.r.t. changes at that input location (e.g., if small changes at a pixel can have a large effect on the network output then it is logical to assume that the network is “paying attention” to that pixel). Let’s define the gradient of the loss w.r.t input for teacher and student as:
JS = ∂
∂x L(WS, x), JT =
∂
∂x L(WT, x) (3)
Then if we want student gradient attention to be similar to teacher attention, we can minimize a distance between them (here we use l2 distance but other distances can be employed as well):
LAT (WS,WT, x) = L(WS, x) + β
2 ||JS − JT ||2 (4)
As WT and x are given, to get the needed derivative w.r.t. WS :
∂
∂WS LAT =
∂
∂WS L(WS, x) + β(JS − JT )
∂2
∂WS∂x L(WS, x) (5)
So to do an update we first need to do forward and back propagation to get JS and JT , compute the second error β2 ||JS − JT ||2 and propagate it second time. The second propagation is similar to forward propagation in this case, and involves second order mixed partial derivative calculation ∂ 2
∂WS∂x .
The above computation is similar to the double backpropagation technique developed by Drucker & LeCun (1992) (where the l2 norm of the gradient w.r.t. input is used as regularizer). Furthermore, it can be implemented efficiently in a framework with automatic differentiation support, even for modern architectures with sophisticated graphs. The second backpropagation has approximately the same cost with first backpropagation, excluding forward propagation.
We also propose to enforce horizontal flip invariance on gradient attention maps. To do that we propagate horizontally flipped images as well as originals, backpropagate and flip gradient attention maps back. We then add l2 losses on the obtained attentions and outputs, and do second backpropagation:
Lsym(W, x) = L(W, x) + β 2 || ∂ ∂x L(W, x)− flip( ∂ ∂x L(W,flip(x)))||2 , (6)
where flip(x) denotes the flip operator. This is similar to Group Equivariant CNN approach by Cohen & Welling (2016), however it is not a hard constraint. We experimentally find that this has a regularization effect on training.
We should note that in this work we consider only gradients w.r.t. the input layer, but in general one might have the proposed attention transfer and symmetry constraints w.r.t. higher layers of the network.
4 EXPERIMENTAL SECTION
In the following section we explore attention transfer on various image classification datasets. We split the section in two parts, in the first we include activation-based attention transfer and gradientbased attention transfer experiments on CIFAR, and in the second activation-based attention trans-
fer experiments on larger datasets. For activation-based attention transfer we used Network-InNetwork Lin et al. (2013) and ResNet-based architectures (including the recently introduced Wide Residual Networks (WRN) Zagoruyko & Komodakis (2016)), as they are most performant and set strong baselines in terms of number of parameters compared to AlexNet or VGG, and have been explored in various papers across small and large datasets. On Scenes, CUB and ImageNet we experimented with ResNet-18 and ResNet-34. As for gradient-based attention, we constrained ourselves to Network-In-Network without batch normalization and CIFAR dataset, due to the need of complex automatic differentiation.
4.1 CIFAR EXPERIMENTS
We start with CIFAR dataset which has small 32 × 32 images, and after downsampling top activations have even smaller resolution, so there is not much space for attention transfer. Interestingly, even under this adversarial setting, we find that attention transfer seems to give reasonable benefits, offering in all cases consistent improvements. We use horizontal flips and random crops data augmentations, and all networks have batch normalization. We find that ZCA whitening has negative effect on validation accuracy, and omit it in favor of simpler meanstd normalization. We raise Knowledge Distillation (KD) temperature for ResNet transfers to 4, and use α = 0.9 (see Hinton et al. (2015) for an explanation of these parameters).
4.1.1 ACTIVATION-BASED ATTENTION TRANSFER
Results of attention transfer (using F 2sum attention maps) for various networks on CIFAR-10 can be found in table 1. We experimented with teacher/student having the same depth (WRN-16-2/WRN16-1), as well as different depth (WRN-40-1/WRN-16-1, WRN-40-2/WRN-16-2). In all combinations, attention transfer (AT) shows significant improvements, which are also higher when it is combined with knowledge distillation (AT+KD).
To verify if having at least one activation-based attention transfer loss per group in WRN transfer is important, we trained three networks with only one transfer loss per network in group1, group2 and group3 separately, and compared to a network trained with all three losses. The corresponding results were 8.11, 7.96, 7.97 (for the separate losses) and 7.93 for the combined loss (using WRN16-2/WRN-16-1 as teacher/student pair). Each loss provides some additional degree of attention transfer.
We also explore which attention mapping functions tend to work best using WRN-16-1 and WRN16-2 as student and teacher networks respectively (table 2). Interestingly, sum-based functions work very similar, and better than max-based ones. From now on, we will use sum of squared attention mapping function F 2sum for simplicity. As for parameter β in eq. 2, it usually varies about 0.1, as we set it to 103 divided by number of elements in attention map and batch size for each layer. In case of combinining AT with KD we decay it during traning in order to simplify learning harder examples.
4.1.2 ACTIVATION-BASED AT VS. TRANSFERRING FULL ACTIVATION
To check if transferring information from full activation tensors is more beneficial than from attention maps, we experimented with FitNets-style hints using l2 losses on full activations directly, with 1 × 1 convolutional layers to match tensor shapes, and found that improvements over baseline student were minimal (see column F-ActT in table 1). For networks of the same width different depth we tried to regress directly to activations, without 1× 1 convolutions. We also use l2 normalization before transfer losses, and decay β in eq. 2 during training as these give better performance. We find that AT, as well as full-activation transfer, greatly speeds up convergence, but AT gives much
better final accuracy improvement than full-activation transfer (see fig. 7(b), Appendix). It seems quite interesting that attention maps carry information that is more important for transfer than full activations.
attention mapping function error no attention transfer 8.77 Fsum 7.99 F 2sum 7.93 F 4sum 8.09 F 1max 8.08
Table 2: Test error of WRN16-2/WRN-16-1 teacher/student pair for various attention mapping functions. Median of 5 runs test errors are reported.
norm type error baseline (no attention transfer) 13.5 min-l2 Drucker & LeCun (1992) 12.5 grad-based AT 12.1 KD 12.1 symmetry norm 11.8 activation-based AT 11.2
Table 3: Performance of various gradient-based attention methods on CIFAR-10. Baseline is a thin NIN network with 0.2M parameters (trained only on horizontally flipped augmented data and without batch normalization), min-l2 refers to using l2 norm of gradient w.r.t. input as regularizer, symmetry norm - to using flip invariance on gradient attention maps (see eq. 6), AT - to attention transfer, and KD - to Knowledge Distillation (both AT and KD use a wide NIN of 1M parameters as teacher).
4.1.3 GRADIENT-BASED ATTENTION TRANSFER
For simplicity we use thin Network-In-Network model in these experiments, and don’t apply random crop data augmentation with batch normalization, just horizontal flips augmentation. We also only use deterministic algorithms and sampling with fixed seed, so reported numbers are for single run experiments. We find that in this setting network struggles to fit into training data already, and turn off weight decay even for baseline experiments. In future we plan to explore gradient-based attention for teacher-student pairs that make use of batch normalization, because it is so far unclear how batch normalization should behave in the second backpropagation step required during gradientbased attention transfer (e.g., should it contribute to batch normalization parameters, or is a separate forward propagation with fixed parameters needed).
We explored the following methods:
• Minimizing l2 norm of gradient w.r.t. input, i.e. the double backpropagation method Drucker & LeCun (1992);
• Symmetry norm on gradient attention maps (see eq. 6);
• Student-teacher gradient-based attention transfer;
• Student-teacher activation-based attention transfer.
Results for various methods are shown in table 3. Interestingly, just minimizing l2 norm of gradient already works pretty well. Also, symmetry norm is one the best performing attention norms, which we plan to investigate in future on other datasets as well. We also observe that, similar to activationbased attention transfer, using gradient-based attention transfer leads to improved performance. We also trained a network with activation-based AT in the same training conditions, which resulted in the best performance among all methods. We should note that the architecture of student NIN without batch normalization is slightly different from teacher network, it doesn’t have ReLU activations before pooling layers, which leads to better performance without batch normalization, and worse with. So to achieve the best performance with activation-based AT we had to train a new teacher, with batch normalization and without ReLU activations before pooling layers, and have AT losses on outputs of convolutional layers.
4.2 LARGE INPUT IMAGE NETWORKS
In this section we experiment with hidden activation attention transfer on ImageNet networks which have 224 × 224 input image size. Presumably, attention matters more in this kind of networks as spatial resolution of attention maps is higher.
4.2.1 TRANSFER LEARNING
To see how attention transfer works in finetuning we choose two datasets: Caltech-UCSD Birds-2002011 fine-grained classification (CUB) by Wah et al. (2011), and MIT indoor scene classification (Scenes) by Quattoni & Torralba (2009), both containing around 5K images training images. We took ResNet-18 and ResNet-34 pretrained on ImageNet and finetuned on both datasets. On CUB we crop bounding boxes, rescale to 256 in one dimension and then take a random crop. Batch normalization layers are fixed for finetuning, and first group of residual blocks is frozen. We then took finetuned ResNet-34 networks and used them as teachers for ResNet-18 pretrained on ImageNet, with F 2sum attention losses on 2 last groups. In both cases attention transfer provides significant improvements, closing the gap between ResNet-18 and ResNet-34 in accuracy. On Scenes AT works as well as KD, and on CUB AT works much better, which we speculate is due to importance of intermediate attention for fine-grained recognition. Moreover, after finetuning, student’s attention maps indeed look more similar to teacher’s (Fig. 6, Appendix).
4.2.2 IMAGENET
To showcase activation-based attention transfer on ImageNet we took ResNet-18 as a student, and ResNet-34 as a teacher, and tried to improve ResNet-18 accuracy. We added only two losses in the 2 last groups of residual blocks and used squared sum attention F 2sum. We also did not have time to tune any hyperparameters and kept them from finetuning experiments. Nevertheless, ResNet-18 with attention transfer achieved 1.1% top-1 and 0.8% top-5 better validation accuracy (Table. 5 and
Fig. 7(a), Appendix), we plan to update the paper with losses on all 4 groups of residual blocks.
We were not able to achieve positive results with KD on ImageNet. With ResNet-18-ResNet-34 student-teacher pair it actually hurts convergence with the same hyperparameters as on CIFAR. As it was reported that KD struggles to work if teacher and student have different architecture/depth (we observe the same on CIFAR), so we tried using the same architecture and depth for attention transfer. On CIFAR both AT and KD work well in this case and improve convergence and final accuracy, on ImageNet though KD converges significantly slower (we did not train until the end due to lack of computational resources). We also could not find applications of FitNets, KD or similar methods on ImageNet in the literature. Given that, we can assume that proposed activation-based AT is the first knowledge transfer method to be successfully applied on ImageNet.
5 CONCLUSIONS
We presented several ways of transferring attention from one network to another, with experimental results over several image recognition datasets. It would be interesting to see how attention transfer works in cases where spatial information is more important, e.g. object detection or weaklysupervised localization, which is something that we plan to explore in the future.
Overall, we think that our interesting findings will help further advance knowledge distillation, and understanding convolutional neural networks in general.
A APPENDIX
A.1 FIGURES AND TABLES
videostore
0 1 2 3 4 5 6 7
0
2
4
6
8
10
bookshop, bookstore, bookstall
0 1 2 3 4 5 6 7
0
2
4
6
8
10
videostore
0 1 2 3 4 5 6 7
0
2
4
6
8
10
videostore
0 1 2 3 4 5 6 7
0
2
4
6
8
10
videostore
hospitalroom
0 2 4 6 8 10
0 1 2 3 4 5 6 7
desk
0 2 4 6 8 10
0 1 2 3 4 5 6 7
hospitalroom
0 2 4 6 8 10
0 1 2 3 4 5 6 7
hospitalroom
0 2 4 6 8 10
0 1 2 3 4 5 6 7
hospitalroom
dentaloffice
0 1 2 3 4 5 6 7 ResNet-18-ImageNet
0
2
4
6
8
10
desk
0 1 2 3 4 5 6 7 ResNet-18-scenes
0
2
4
6
8
10
dentaloffice
0 1 2 3 4 5 6 7 ResNet-18-scenes-AT
0
2
4
6
8
10
dentaloffice
0 1 2 3 4 5 6 7 ResNet-34-scenes
0
2
4
6
8
10
dentaloffice
A.2 IMPLEMENTATION DETAILS
The experiments were conducted in Torch machine learning framework. Double propagation can be implemented in a modern framework with automatic differentiation support, e.g. Torch, Theano, Tensorflow. For ImageNet experiments we used fb.resnet.torch code, and used 2 Titan X cards with data parallelizm in both teacher and student to speed up training. Code and models for our experiments are available at https://github.com/szagoruyko/attention-transfer.

This paper proposes to investigate attention transfers between a teacher and a student network.
Attention transfer is performed by minimising the l2 distance between the teacher/student attention maps at different layers, in addition to minimising the classification loss and optionally a knowledge distillation term.
Authors define several activation based attentions (sum of absolute feature values raise at the power p or max of values raised at the power p). They also propose a gradient based attention (derivative of the Loss w.r.t. inputs).
They evaluate their approaches on several datasets (CIFAR, Cub/Scene, Imagenet) showing that attention transfers  does help improving the student network test performance.  However, the student networks performs worst than the teacher, even with attention.
Few remarks/questions:
- in section 3 authors  claim that networks with higher accuracy have a higher spatial correlation between the object and the attention map. While Figure 4 is compelling, it would be nice to have quantitative results showing that as well.
- how did you choose the hyperparameter values, it would be nice to see what is the impact of $\beta$.
- it would be nice to report teacher train and validation loss in Figure 7 b)
- from the experiments, it is not clear what at the pros/cons of the different attention maps
- AT does not lead to better result than the teacher. However, the student networks have less parameters. It would be interesting to characterise the corresponding speed-up. If you keep the same architecture between the student and the teacher, is there any benefit to the attention transfer?
In summary:
Pros:
- Clearly written and well motivated.
- Consistent improvement of the student with attention compared to the student alone.
Cons:
- Students have worst performances than the teacher models.
- It is not clear which attention to use in which case?
- Somewhat incremental novelty relatively to Fitnet


SKIP-GRAM NEGATIVE SAMPLING
Skip-Gram Negative Sampling (SGNS) word embedding model, well known by its implementation in “word2vec” software, is usually optimized by stochastic gradient descent. It can be shown that optimizing for SGNS objective can be viewed as an optimization problem of searching for a good matrix with the low-rank constraint. The most standard way to solve this type of problems is to apply Riemannian optimization framework to optimize the SGNS objective over the manifold of required low-rank matrices. In this paper, we propose an algorithm that optimizes SGNS objective using Riemannian optimization and demonstrates its superiority over popular competitors, such as the original method to train SGNS and SVD over SPPMI matrix.
1 INTRODUCTION
In this paper, we consider the problem of embedding words into a low-dimensional space in order to measure the semantic similarity between them. As an example, how to find whether the word “table” is semantically more similar to the word “stool” than to the word “sky”? That is achieved by constructing a low-dimensional vector representation for each word and measuring similarity between the words as the similarity between the corresponding vectors.
One of the most popular word embedding models by Mikolov et al. (2013) is a discriminative neural network that optimizes Skip-Gram Negative Sampling (SGNS) objective (see Equation 3). It aims at predicting whether two words can be found close to each other within a text. As shown in Section 2, the process of word embeddings training using SGNS can be divided into two general steps with clear objectives:
Step 1. Search for a low-rank matrix X that provides a good SGNS objective value; Step 2. Search for a good low-rank representation X = WC⊤ in terms of linguistic metrics,
where W is a matrix of word embeddings and C is a matrix of so-called context embeddings.
Unfortunately, most previous approaches mixed these two steps into a single one, what entails a not completely correct formulation of the optimization problem. For example, popular approaches to train embeddings (including the original “word2vec” implementation) do not take into account that the objective from Step 1 depends only on the product X = WC⊤: instead of straightforward computing of the derivative w.r.t. X , these methods are explicitly based on the derivatives w.r.t. W and C, what complicates the optimization procedure. Moreover, such approaches do not take into account that parametrization WC⊤ of matrix X is non-unique and Step 2 is required. Indeed, for any invertible matrix S, we have X = W1C⊤1 = W1SS −1C⊤1 = W2C ⊤ 2 , therefore, solutions W1C1 and W2C2 are equally good in terms of the SGNS objective but entail different cosine similarities between embeddings and, as a result, different performance in terms of linguistic metrics (see Section 4.2 for details).
A successful attempt to follow the above described steps, which outperforms the original SGNS optimization approach in terms of various linguistic tasks, was proposed by Levy & Goldberg (2014). In order to obtain a low-rank matrix X on Step 1, the method reduces the dimensionality of Shifted Positive Pointwise Mutual Information (SPPMI) matrix via Singular Value Decomposition (SVD). On Step 2, it computes embeddings W and C via a simple formula that depends on the factors obtained by SVD. However, this method has one important limitation: SVD provides a solution to a surrogate optimization problem, which has no direct relation to the SGNS objective. In fact, SVD minimizes the Mean Squared Error (MSE) between X and SPPMI matrix, what does not lead to minimization of SGNS objective in general (see Section 6.1 and Section 4.2 in Levy & Goldberg (2014) for details).
These issues bring us to the main idea of our paper: while keeping the low-rank matrix search setup on Step 1, optimize the original SGNS objective directly. This leads to an optimization problem over matrixX with the low-rank constraint, which is often (Mishra et al. (2014)) solved by applying Riemannian optimization framework (Udriste (1994)). In our paper, we use the projector-splitting algorithm (Lubich & Oseledets (2014)), which is easy to implement and has low computational complexity. Of course, Step 2 may be improved as well, but we regard this as a direction of future work.
As a result, our approach achieves the significant improvement in terms of SGNS optimization on Step 1 and, moreover, the improvement on Step 1 entails the improvement on Step 2 in terms of linguistic metrics. That is why, the proposed two-step decomposition of the problem makes sense, what, most importantly, opens the way to applying even more advanced approaches based on it (e.g., more advanced Riemannian optimization techniques for Step 1 or a more sophisticated treatment of Step 2).
To summarize, the main contributions of our paper are:
• We reformulated the problem of SGNS word embedding learning as a two-step procedure with clear objectives;
• For Step 1, we developed an algorithm based on Riemannian optimization framework that optimizes SGNS objective over low-rank matrix X directly;
• Our algorithm outperforms state-of-the-art competitors in terms of SGNS objective and the semantic similarity linguistic metric (Levy & Goldberg (2014); Mikolov et al. (2013); Schnabel et al. (2015)).
2 PROBLEM SETTING
2.1 SKIP-GRAM NEGATIVE SAMPLING
In this paper, we consider the Skip-Gram Negative Sampling (SGNS) word embedding model (Mikolov et al. (2013)), which is a probabilistic discriminative model. Assume we have a text corpus given as a sequence of words w1, . . . , wn, where n may be larger than 1012 and wi ∈ VW belongs to a vocabulary of words VW . A context c ∈ VC of the word wi is a word from set {wi−L, ..., wi−1, wi+1, ..., wi+L} for some fixed window size L. Letw, c ∈ Rd be the word embeddings of word w and context c, respectively. Assume they are specified by the following mappings:
W : VW → Rd, C : VC → Rd. The ultimate goal of SGNS word embedding training is to fit good mappingsW and C. In the SGNS model, the probability that pair (w, c) is observed in the corpus is modeled as a following function:
P ((w, c) ∈ D|w, c) = σ(⟨w, c⟩) = 1 1 + exp(−⟨w, c⟩) , (1)
where D is the multiset of all word-context pairs (w, c) observed in the corpus and ⟨x,y⟩ is the scalar product of vectors x and y. Number d is a hyperparameter that adjusts the flexibility of the model. It usually takes values from tens to hundreds.
In order to collect a training set, we take all pairs (w, c) fromD as positive examples and k randomly generated pairs (w, c) as negative ones. Let #(w, c) be the number of times the pair (w, c) appears
in D. Thereby the number of times the word w and the context c appear in D can be computed as #(w) = ∑ c∈Vc #(w, c) and #(c) = ∑ w∈Vw #(w, c) accordingly. Then negative examples are generated from the distribution defined by #(c) counters: PD(c) = #(c) |D| . In this way, we have a model maximizing the following logarithmic likelihood objective for each word pair (w, c):
#(w, c)(log σ(⟨w, c⟩) + k · Ec′∼PD log σ(−⟨w, c′⟩)). (2) In order to maximize the objective over all observations for each pair (w, c), we arrive at the following SGNS optimization problem over all possible mappingsW and C:
l = ∑
w∈VW ∑ c∈VC #(w, c)(log σ(⟨w, c⟩) + k · Ec′∼PD log σ(−⟨w, c′⟩)) → maxW, . (3)
Usually, this optimization is done via the stochastic gradient descent procedure that is performed during passing through the corpus (Mikolov et al. (2013); Rong (2014)).
2.2 OPTIMIZATION OVER LOW-RANK MATRICES
Relying on the prospect proposed by Levy & Goldberg (2014), let us show that the optimization problem given by (3) can be considered as a problem of searching for a matrix that maximizes a certain objective function and has the rank-d constraint (Step 1 in the scheme described in Section 1).
2.2.1 SGNS LOSS FUNCTION
As shown by Levy & Goldberg (2014), the logarithmic likelihood (3) can be represented as the sum of lw,c(w, c) over all pairs (w, c), where lw,c(w, c) has the following form:
lw,c(w, c) =#(w, c) log σ(⟨w, c⟩) + k #(w)#(c)
|D| log σ(−⟨w, c⟩). (4)
A crucial observation is that this loss function depends only on the scalar product ⟨w, c⟩ but not on embeddings w and c separately:
lw,c(w, c) = fw,c(xw,c),
fw,c(xw,c) = aw,c log σ(xw,c) + bw,c log σ(−xw,c), where xw,c is the scalar product ⟨w, c⟩ and aw,c = #(w, c), bw,c = k#(w)#(c)|D| are constants.
2.2.2 MATRIX NOTATION
Denote |VW | as n and |VC | as m. Let W ∈ Rn×d and C ∈ Rm×d be matrices, where each row w ∈ Rd of matrix W is the word embedding of the corresponding word w and each row c ∈ Rd of matrix C is the context embedding of the corresponding context c. Then the elements of the product of these matrices
X = WC⊤
are the scalar products xw,c of all pairs (w, c):
X = (xw,c), w ∈ VW , c ∈ VC . Note that this matrix has rank d, becauseX equals to the product of two matrices with sizes (n× d) and (d×m). Now we can write SGNS objective given by (3) as a function of X:
F (X) = ∑
w∈VW ∑ c∈VC fw,c(xw,c), F : Rn×m → R. (5)
This arrives us at the following proposition:
Proposition 1 SGNS optimization problem given by (3) can be rewritten in the following constrained form:
maximize X∈Rn×m F (X), subject to X ∈ Md, (6)
whereMd is the manifold (Udriste (1994)) of all matrices in Rn×m with rank d: Md = {X ∈ Rn×m : rank(X) = d}.
The key idea of this paper is to solve the optimization problem given by (6) via the framework of Riemannian optimization, which we introduce in Section 3.
Important to note that this prospect does not suppose the optimization over parameters W and C directly. This entails the optimization in the space with ((n + m − d) · d) degrees of freedom (Mukherjee et al. (2015)) instead of ((n + m) · d), what simplifies the optimization process (see Section 5 for the experimental results).
2.3 COMPUTING EMBEDDINGS FROM A LOW-RANK SOLUTION
OnceX is found, we need to recoverW andC such thatX = WC⊤ (Step 2 in the scheme described in Section 1). This problem does not have a unique solution, since if (W,C) satisfy this equation, thenWS−1 and CS⊤ satisfy it as well for any non-singular matrix S. Moreover, different solutions may achieve different values of the linguistic metrics (see Section 4.2 for details). While our paper focuses on Step 1, we use, for Step 2, a heuristic approach that was proposed by Levy et al. (2015) and it shows good results in practice. We compute SVD of X in the form X = UΣV ⊤, where U and V have orthonormal columns, and Σ is the diagonal matrix, and use
W = U √ Σ, C = V √ Σ
as matrices of embeddings.
A simple justification of this solution is the following: we need to map words into vectors in a way that similar words would have similar embeddings in terms of cosine similarities:
cos(w1,w2) = ⟨w1,w2⟩
∥w1∥ · ∥w2∥ .
It is reasonable to assume that two words are similar, if they share contexts. Therefore, we can estimate the similarity of two words w1, w2 as s(w1, w2) = ∑ c∈VC xw1,c · xw2,c, what is the element of the matrix XX⊤ with indices (w1, w2). Note that XX⊤ = UΣV ⊤V ΣU⊤ = UΣ2U⊤. If we choose W = UΣ, we exactly obtain ⟨w1,w2⟩ = s(w1, w2), since WW⊤ = XX⊤ in this case. That is, the cosine similarity of the embeddingsw1,w2 coincides with the intuitive similarity s(w1, w2). However, scaling by √ Σ instead of Σ was shown by Levy et al. (2015) to be a better solution in experiments.
3 PROPOSED METHOD
3.1 RIEMANNIAN OPTIMIZATION
3.1.1 GENERAL SCHEME
The main idea of Riemannian optimization (Udriste (1994)) is to consider (6) as a constrained optimization problem. Assume we have an approximated solution Xi on a current step of the optimization process, where i is the step number. In order to improve Xi, the next step of the standard gradient ascent outputs Xi + ∇F (Xi), where ∇F (Xi) is the gradient of objective F at the point Xi. Note that the gradient ∇F (Xi) can be naturally considered as a matrix in Rn×m. Point Xi + ∇F (Xi) leaves the manifold Md, because its rank is generally greater than d. That is why Riemannian optimization methods map point Xi + ∇F (Xi) back to manifold Md. The standard Riemannian gradient method first projects the gradient step onto the tangent space at the current point Xi and then retracts it back to the manifold:
Xi+1 = R (PTM (Xi +∇F (Xi))),
where R is the retraction operator, and PTM is the projection onto the tangent space.
3.1.2 PROJECTOR-SPLITTING ALGORITHM
In our paper, we use a much simpler version of such approach that retracts point Xi + ∇F (Xi) directly to the manifold, as illustrated on Figure 1: Xi+1 = R(Xi +∇F (Xi)).
Under review as a conference paper at ICLR 2017
Fine-tuning word embeddings xxxxx xxxxx xxxxx xxxx xxxx xxxx xxx xxxxx xxxx xxxxx ABSTRACT Blah-blah
Keywords word embeddings, SGNS, word2vec, GLOVE
1. INTRODUCTION sdfdsf
2. CONCLUSIONS
3. RELATED WORK Mikolov main [?] Levi main [?]
rFi
Xi = UiSiV T i
Xi+1 = Ui+1Si+1V T i+1
retraction
4. CONCLUSIONS
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. WOODSTOCK ’97 El Paso, Texas USA Copyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$15.00.
Fine-tuning word embeddings xxxxx xxxxx xxxxx xxxx xxxx xxxx xxx xxxxx xxxx xxxxx ABSTRACT Blah-blah Keywords word embeddings, SGNS, word2vec, GLOVE
1. INTRODUCTION sdfdsf
2. CONCLUSIONS
3. RELATED WORK Mikolov main [?] Levi main [?]
rFi
Xi = UiSiV T i
Xi+1 = Ui+1Si+1V T i+1
retraction
Md
4. CONCLUSIONS
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. WOODSTOCK ’97 El Paso, Texas USA Copyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$15.00.
Fine-tuning word embeddings xxxxx xxxxx xxxxx xxxx xxxx xxxx xxx xxxxx xxxx xxxxx
ABSTRACT Blah-blah
Keywords word embeddings, SGNS, word2vec, GLOVE
1. INTRODUCTION sdfdsf
2. CONCLUSIONS
3. RELATED WORK Mikolov main [?] Levi main [?]
rF (Xi)
Xi +rF (Xi)
Xi = UiSiV T i
Xi
Xi+1
Xi+1 = Ui+1Si+1V T i+1
retraction
Md
4. CONCLUSIONS
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. WOODSTOCK ’97 El Paso, Texas USA Copyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$15.00.
Fine-tuning word embeddings xxxxx xxxxx xxxxx xxx xxxx xxxx xxx xxxxx xxxx xxxxx ABSTRACT Blah-blah
Keywords word embeddings, SGNS, word2vec, GLOVE
1. INTRODUCTION
sdfdsf
2. CONCLUSIONS
3. RELATED WORK Mikolov main [?] Levi main [?]
rF (Xi)
Xi +rF (Xi)
Xi = UiSiV T i
Xi
Xi+1
Xi+1 = Ui+1Si+1V T i+1
retraction
Md
4. CONCLUSIONS
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. WOODSTOCK ’97 El Paso, Texas USA Copyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$15.00.
Fine-tuning word embeddings xxxxx xxxxx xxxxx xxxx xxxx xxxx xxx xxxxx xxxx xxxxx
ABSTRACT Blah-blah
Keywords word embeddings, SGNS, word2vec, GLOVE
1. INTRODUCTION sdfdsf
2. CONCLUSIONS
3. RELATED WORK Mikolov main [?] Levi main [?]
rF (Xi)
Xi +rF (Xi)
Xi = UiSiV T i
Xi
Xi+1
Xi+1 = Ui+1Si+1V T i+1
retraction
Md
4. CONCLUSIONS
Permission to make digital or hard copies of all or part of this work for personal or classro m us is r nted without f e provided tha copies are not made or distributed for profit or commercial advantage an that copies bear this notice and the full citation n the first page. To copy otherwise, to republish, to post on servers or to redistribute o lists, requires prior specific permission and/or a fee. WOODSTOCK ’97 El Paso, Texas USA Copyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$15.00.
Fine-tuning word embeddings xxxxx xx xxxxx xxxx xxxx xxxx xxx xxxxx xxxx xxxxx ABSTRACT Blah-blah
Keywords word embeddings, SGNS, word2vec, GLOVE
1. INT ODUCTION sdfdsf
2. CONCLUSIONS
3. RELATED WORK Mikolov main [?] Levi main [?]
rF (Xi)
Xi +rF (Xi)
Xi = UiSiV T i
Xi
Xi+1
Xi+1 = Ui+1Si+1V T i+1
retraction
Md
4. CONCLUSIONS
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. WOODSTOCK ’97 El Paso, Texas USA Copyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$15.00.
Figure 1: Geometric interpretation of one step of projector-splitting optimization procedure: the gradient step an the retraction of the high-rank atr x Xi + ∇F (Xi) to the manifold of low-rank matrices Md.
Intuitively, retractor R finds rank-d matrix on t e manifold Md tha is similar to high-rank matrixXi+∇F (Xi) in terms o Frobeniu orm. How can e do it? The most straightforward way to reduce the rank ofXi +∇F (Xi) is to perform the SVD, which keeps d largest singular values of it:
1: Ui+1, Si+1, V ⊤i+1 ← SVD(Xi +∇F (Xi)), 2: Xi+1 ← Ui+1Si+1V ⊤i+1.
(7)
However, it is computationally expensive. Instead of this approach, we use the projector-splitting method (Lubich & Oseledets (2014)), which is a second-order retraction onto the manifold (for details, see the review by Absil & Oseledets (2015)). Its practical implementation is also quite intuitive: instead of computing the full SVD of Xi +∇F (Xi) according to the gradient projection method, we use just one step of the block power numerical method (Bentbib & Kanber (2015)) which computes the SVD, what reduces the computational complexity.
Let us keep the current point in the following factorized form: Xi = UiSiV ⊤ i , (8) where matrices Ui ∈ Rn×d and Vi ∈ Rm×d have d orthonormal columns and Si ∈ Rd×d. Then we need to perform two QR-decompositions to retract point Xi +∇F (Xi) back to the manifold:
1: Ui+1, Si+1 ← QR ((Xi +∇F (Xi))Vi) , 2: Vi+1, S⊤i+1 ← QR ( (Xi +∇F (Xi))⊤Ui+1 ) ,
3: Xi+1 ← Ui+1Si+1V ⊤i+1. In this way, we always keep the solution Xi+1 = Ui+1Si+1V ⊤i+1 on the manifold Md and in the form (8).
What is important, we only need to compute ∇F (Xi), so the gradients with respect to U , S and V are never computed explicitly, thus avoiding the subtle case where S is close to singular (so-called singular (critical) point on the manifold). Indeed, the gradient with respect to U (while keeping the orthogonality constraints) can be written (Koch & Lubich (2007)) as:
∂F ∂U = ∂F ∂X V S−1,
which means that the gradient will be large if S is close to singular. The projector-splitting scheme is free from this problem.
3.2 ALGORITHM
In case of SGNS objective given by (5), an element of gradient ∇F has the form:
(∇F (X))w,c = ∂fw,c(xw,c)
∂xw,c = #(w, c) · σ (−xw,c)− k
#(w)#(c)
|D| · σ (xw,c) .
To make the method more flexible in terms of convergence properties, we additionally use λ ∈ R, which is a step size parameter. In this case, retractor R returns Xi + λ∇F (Xi) instead of Xi +∇F (Xi) onto the manifold. The whole optimization procedure is summarized in Algorithm 1.
Algorithm 1 Riemannian Optimization for SGNS Require: Dimentionality d, initializationW0 and C0, step size λ, gradient function∇F : Rn×m →
Rn×m, number of iterationsK Ensure: FactorW ∈ Rn×d 1: X0 ← W0C⊤0 # get an initial point at the manifold 2: U0, S0, V ⊤0 ← SVD(X0) # compute the first point satisfying the low-rank constraint 3: i ← 0 4: while i < K do 5: Ui+1, Si+1 ← QR ((Xi + λ∇F (Xi))Vi) # perform one step of the block power method
with two QR-decompositions 6: Vi+1, S⊤i+1 ← QR ( (Xi + λ∇F (Xi))⊤Ui+1 ) 7: Xi+1 ← Ui+1Si+1V ⊤i+1 # update the point at the manifold 8: i ← i+ 1 9: end while 10: U,Σ, V ⊤ ← SVD(XK) 11: W ← U √ Σ # compute word embeddings 12: return W
4 EXPERIMENTAL SETUP
4.1 TRAINING MODELS
We compare our method (“RO-SGNS” in the tables) performance to two baselines: SGNS embeddings optimized via Stochastic Gradient Descent, implemented in the original “word2vec”, (“SGDSGNS” in the tables) by Mikolov et al. (2013) and embeddings obtained by SVD over SPPMI matrix (“SVD-SPPMI” in the tables) by Levy & Goldberg (2014). We have also experimented with the blockwise alternating optimization over factors W and C, but the results are almost the same to SGD results, that is why we do not to include them into the paper. The source code of our experiments is available online1.
The models were trained on English Wikipedia “enwik9” corpus2, which was previously used in most papers on this topic. Like in previous studies, we counted only the words which occur more than 200 times in the training corpus (Levy & Goldberg (2014); Mikolov et al. (2013)). As a result, we obtained a vocabulary of 24292 unique tokens (set of words VW and set of contexts VC are equal). The size of the context window was set to 5 for all experiments, as it was done by Levy & Goldberg (2014); Mikolov et al. (2013). We conduct two series of experiments: for dimensionality d = 100 and d = 200.
Optimization step size is chosen to be small enough to avoid huge gradient values. However, thorough choice of λ does not result in a significant difference in performance (this parameter was tuned on the training data only, the exact values used in experiments are reported below).
4.2 EVALUATION
We evaluate word embeddings via the word similarity task. We use the following popular datasets for this purpose: “wordsim-353” (Finkelstein et al. (2001); 3 datasets), “simlex-999” (Hill et al. (2016)) and “men” (Bruni et al. (2014)). Original “wordsim-353” dataset is a mixture of the word pairs for both word similarity and word relatedness tasks. This dataset was split (Agirre et al. (2009)) into two intersecting parts: “wordsim-sim” (“ws-sim” in the tables) and “wordsim-rel” (“ws-rel” in the tables) to separate the words from different tasks. In our experiments, we use both of them on a par with the full version of “wordsim-353” (“ws-full” in the tables). Each dataset contains word pairs together with assessor-assigned similarity scores for each pair. As a quality measure, we use Spearman’s correlation between these human ratings and cosine similarities for each pair. We call this quality metric linguistic in our paper.
1https://github.com/newozz/riemannian_sgns 2Enwik9 corpus can be found here: http://mattmahoney.net/dc/textdata
5 RESULTS OF EXPERIMENTS
First of all, we compare the value of SGNS objective obtained by the methods. The comparison is demonstrated in Table 1.
We see that SGD-SGNS and SVD-SPPMI methods provide quite similar results, however, the proposed method obtains significantly better SGNS values, what proves the feasibility of using Riemannian optimization framework in SGNS optimization problem. It is interesting to note that SVDSPPMI method, which does not optimize SGNS objective directly, obtains better results than SGDSGNS method, which aims at optimizing SGNS. This fact additionally confirms the idea described in Section 2.2.2 that the independent optimization over parameters W and C may decrease the performance.
However, the target performance measure of embedding models is the correlation between semantic similarity and human assessment (Section 4.2). Table 2 presents the comparison of the methods in terms of it. We see that our method outperforms the competitors on all datasets except for “men” dataset where it obtains slightly worse results. Moreover, it is important that the higher dimension entails higher performance gain of our method in comparison to the competitors.
In order to understand how exactly our model improves or degrades the performance in comparison to the baseline, we found several words, whose neighbors in terms of cosine distance change significantly. Table 3 demonstrates neighbors of words “five”, “he” and “main” in terms of our model and its nearest competitor according to the similarity task — SVD-SPPMI. These words were chosen as representative examples whose neighborhoods in terms of SVD-SPPMI and RO-SGNS models are strikingly different. A neighbour of a source word is bold if we suppose that it has a similar semantic meaning to the source word. First of all, we notice that our model produces much better neighbors of the words describing digits or numbers (see word “five” as an example). The similar situation happens for many other words, e.g. in case of word “main” — the nearest neighbors contain 4 similar words in case of our model instead of 2 in case of SVD-SPPMI. The neighbourhood of word “he” contains less semantically similar words in case of our model. However, it filters out completely irrelevant words, such as “promptly” and “dumbledore”.
Talking about the optimal number K of iterations in the optimization procedure and step size λ, we found that they depend on the particular value of dimensionality d. For d = 100, we have K = 25, λ ≈ 5 · 10−5, and for d = 200, we have K = 13, λ = 10−4. Moreover, it is interesting that the best results were obtained when SVD-SPPMI embeddings were used as an initialization of Riemannian optimization process.
6 RELATED WORK
6.1 WORD EMBEDDINGS
Skip-Gram Negative Sampling was introduced by Mikolov et al. (2013). The “negative sampling” approach was thoroughly described by Goldberg & Levy (2014), and the learning method is ex-
plained by Rong (2014). There are several open-source implementations of SGNS neural network, which is widely known as “word2vec” 34.
As shown in Section 2.2, Skip-Gram Negative Sampling optimization can be reformulated as a problem of searching for a low-rank matrix. In order to be able to use out-of-the-box SVD for this task, Levy & Goldberg (2014) used the surrogate version of SGNS as the objective function. There are two general assumptions made in their algorithm that distinguish it from the SGNS optimization:
1. SVD optimizes Mean Squared Error (MSE) objective instead of SGNS loss function.
2. In order to avoid infinite elements in SPMI matrix, it is transformed in ad-hoc manner (SPPMI matrix) before applying SVD.
This makes the objective not interpretable in terms of the original task (3). As mentioned by Levy & Goldberg (2014), SGNS objective weighs different (w, c) pairs differently, unlike the SVD, which works with the same weight for all pairs, what may entail the performance fall. The comprehensive explanation of the relation between SGNS, SPPMI, SVD-over-SPPMI methods is provided by Keerthi et al. (2015). Lai et al. (2015); Levy et al. (2015) give a good overview of highly practical methods to improve these word embedding models.
6.2 RIEMANNIAN OPTIMIZATION
An introduction to optimization over Riemannian manifolds can be found in the paper of Udriste (1994). The overview of retractions of high rank matrices to low-rank manifolds is provided by Absil & Oseledets (2015). The projector-splitting algorithm was introduced by Lubich & Oseledets (2014), and also was mentioned by Absil & Oseledets (2015) as “Lie-Trotter retraction”.
Riemannian optimization is succesfully applied to various data science problems: for example, matrix completion (Vandereycken (2013)), large-scale recommender systems (Tan et al. (2014)), and tensor completion (Kressner et al. (2014)).
7 CONCLUSIONS AND FUTURE WORK
In our paper, we proposed the general two-step scheme of training SGNS word embedding model and introduced the algorithm that performs the search of a solution in the low-rank form via Riemannian optimization framework. We also demonstrated the superiority of the proposed method, by providing the experimental comparison to the existing state-of-the-art approaches.
It seems to be an interesting direction of future work to apply more advanced optimization techniques to Step 1 of the scheme proposed in Section 1 and to explore the Step 2 — obtaining embeddings with a given low-rank matrix.
3Original Google word2vec: https://code.google.com/archive/p/word2vec/ 4Gensim word2vec: https://radimrehurek.com/gensim/models/word2vec.html

Dear authors,
The authors' response clarified some of my confusion. But I still have the following question:
-- The response said a first contribution is a different formulation: you divide the word embedding learning into two steps, step 1 looks for a low-rank X (by Riemannian optimization), step 2 factorizes X into two matrices (W, C). You are claiming that your model outperforms previous approaches that directly optimizes over (W, C). But since the end result (the factors) is the same, can the authors provide some intuition and justification why the proposed method works better?
As far as I can see, though parameterized differently, the first step of your method and previous methods (SGD) are both optimizing over low-rank matrices. Admittedly, Riemannian optimization avoids the rotational degree of freedom (the invertible matrix S you are mentioning in sec 2.3), but I am not 100% certain at this point this is the source of your gain; learning curves of objectives would help to see if Riemannian optimization is indeed more effective.
-- Another detail I could not easily find is the following. You said a disadvantage of other approaches is that their factors W and C do not directly reflect similarity. Did you try to multiply the factors W and C from other optimizers and then factorize the product using the method in section 2.3, and use the new W for your downstream tasks? I am not sure if this would cause much difference in the performance.
Overall, I think it is always interesting to apply advanced optimization techniques to machine learning problems. The current paper would be stronger from the machine learning perspective, if more thorough comparison and discussion (as mentioned above) are provided. On the other hand, my expertise is not in NLP and I leave it to other reviewers to decide the significance in experimental results.