EFFICIENT TRAINING ON VERY LARGE CORPORA VIA GRAMIAN ESTIMATION
We study the problem of learning similarity functions over very large corpora using neural network embedding models. These models are typically trained using SGD with random sampling of unobserved pairs, with a sample size that grows quadratically with the corpus size, making it expensive to scale. We propose new efficient methods to train such models without sampling unobserved pairs. Inspired by matrix factorization, our approach relies on adding a global quadratic penalty and expressing this term as the inner-product of two Gramians. We show that the gradient of this term can be efficiently computed by maintaining estimates of the Gramians, and develop variance reduction schemes to improve the quality of the estimates. We conduct large-scale experiments that show a significant improvement in training time and generalization performance compared to sampling methods.
Summary of the paper:
This work presents a novel method for similarity function learning using non-linear model. The main problem with the similarity function learning models is the pairwise component of the loss function which grows quadratically with the training set. The existing stochastic approximations which are agnostic to training set size have high variance and this in-turn results in poor convergence and generalisation. This paper presents a new stochastic approximation of the pairwise loss with reduced variance. This is achieved by exploiting the dot-product structure of the least-squares loss and is computationally efficient provided the embedding dimensions are small. The core idea is to rewrite the least-squares as the matrix dot product of two PSD matrices (Grammian). The Grammian matrix is the sum of the outer-product of embeddings along the training samples. The authors present two algorithms for training the model, 1)SAGram: By maintaining a cache of all embedding vectors of training points (O(nk) space)$, whenever a point is encountered it's cache is replaced with it's embedding vector. 2) SOGram: This algorithm keeps a moving average of the Grammian estimate to reduce the variance. Experimental results shows that this approach reduces the variance in the Grammian estimates, results in faster convergence and better generalisation.
Review:
The paper is well written with clear contribution to the problem of similarity  learning.  My only complain is that, I think the evaluation is a bit weak and does not support the claim that is applicable all kinds of problems e.g. nlp and recommender systems. This task in Wikipedia does not seem to be standard (kind of arbitrary) — there are some recommendation results in the appendix but I think it should have been in the main paper.
Overall interesting but I would recommend evaluating in standard similarity learning for nlp and other tasks (perhaps more than one)
There are specific similarity evaluation sets for word embeddings. It can be found in following papers: https://arxiv.org/pdf/1301.3781.pdf  
http://www.aclweb.org/anthology/D15-1036