Context Selection for Embedding Models
Word embeddings are an effective tool to analyze language. They have been recently extended to model other types of data beyond text, such as items in recommendation systems. Embedding models consider the probability of a target observation (a word or an item) conditioned on the elements in the context (other words or items). In this paper, we show that conditioning on all the elements in the context is not optimal. Instead, we model the probability of the target conditioned on a learned subset of the elements in the context. We use amortized variational inference to automatically choose this subset. Compared to standard embedding models, this method improves predictions and the quality of the embeddings.
The authors propose an extension to the Exponential Family Embeddings (EFE) model for producing low dimensional representations of graph data based on its context (EFE extends word2vec-style word embedding models to other data types such as counts or real number by using embedding-context scores to produce the natural parameters of various exponential family distributions). They note that while context-based embedding models have been extensively researched, some contexts are more relevant than others for predicting a given target and informing its embedding. 
This observation has been made for word embeddings in prior work, with [1] using a learned attention mechanism to form a weighted average of predictive token contexts and [2] learning part-of-speech-specific classifiers to produce context weights. Citations to this related work should be added to the paper. There has also been prior work that learns fixed position-dependent weights for each word embedding context, but I am not able to recall the exact citation.
The authors propose a significantly different model to this prior work, however, and use a Bayesian model with latent binary masks to choose the relevant context vectors, which they infer with an amortized neural variational inference method. Several technical novelties are introduced to deal with the sparsity of relevant contexts and the difficulties with discrete latent variables and variably-sized element selection. 
They show that their method gives significant improvements in held-out pseudolikelihood on several exponential family embedding tasks, as well as small improvements in unsupervised discovery of movie genres. They also demonstrate qualitatively that the learned variational posterior over relevant contexts makes sensible predictions with an examination of chosen contexts on a dataset of Safeway food products.
The idea of the paper is a straightforward, intuitively appealing, clear improvement over the standard EFE model, and the technical challenges presented by inference and their solutions will be interesting to practitioners using large-scale variational models.
One interesting technical contribution is the architecture of the amortized inference network, which must deal with variable-sized contexts to predict a posterior over each latent Bernoulli masking variable, and solves the problem of variable-sized contexts using soft binning with Gaussian kernels.
Another is the use of posterior regularization to enforce a sparsity penalty on the selected variables. Presumably, because the Bernoulli priors on the mask variables are independent, they are insufficient to enforce the sort of "spiky" sparsity that we want in context selection, and they show significant improvements to held out likelihood by varying the posterior regularization on effectively the average number of selected contexts. Perhaps the authors would like to more explicitly comment on the trade-off between posterior regularization and prior tuning to obtain the desired sparsity.
Overall, while the evaluation in the paper is fairly limited, the paper presents a clear improvement to the Exponential Family Embeddings model with a sound probabilistic model and inference techniques that will be of interest to practitioners.
1. Ling et al. Not All Contexts Are Created Equal: Better Word Representations with Variable Attention. 2015
2. Liu et al. Part-of-Speech Relevance Weights for Learning Word Embeddings. 2016