Exponential Family Embeddings
Word embeddings are a powerful approach for capturing semantic similarity among terms in a vocabulary. In this paper, we develop exponential family embeddings, a class of methods that extends the idea of word embeddings to other types of high-dimensional data. As examples, we studied neural data with real-valued observations, count data from a market basket analysis, and ratings data from a movie recommendation system. The main idea is to model each observation conditioned on a set of other observations. This set is called the context, and the way the context is defined is a modeling choice that depends on the problem. In language the context is the surrounding words; in neuroscience the context is close-by neurons; in market basket data the context is other items in the shopping cart. Each type of embedding model defines the context, the exponential family of conditional distributions, and how the latent embedding vectors are shared across data. We infer the embeddings with a scalable algorithm based on stochastic gradient descent. On all three applications-neural activity of zebrafish, users' shopping behavior, and movie ratings-we found exponential family embedding models to be more effective than other types of dimension reduction. They better reconstruct held-out data and find interesting qualitative structure.
The paper deals with the general problem of finding latent embeddings for a variety of problems and places the problem in a Bayesian framework by showing that embedding problems from several different applications can be expressed as inference in an exponential family scenario. In this case each observation has both an embedding and a context vector. This framework is defined by three main choices (1) defining the context of an observation, (2) deciding how the conditional probability of an observed data point is linked to its context and (3) how the latent embeddings and context vectors are shared across different observations. Inferring the embeddings and context vector can then be done via stochastic gradient descent fairly simply due to properties of the gradient when dealing with exponential families The generality of this framework is then demonstrating that four different real-life embeddings problems can be formulated in this way. The three choices are addressed for each of the four and a gradient formula is derived for each. In the case of two of the scenarios (real valued neural data and Poisson observations for shopping/movies) the EF-EMB method is applied to some real life data sets and empirically strong results are shown in terms of likelihood estimations/errors. Moreover, some quantitative results on these datasets are given to show how having both a context vector and latent embedding can help in the interpretation of results in these contexts. 
Technical quality: 3   The experimental method was well motivated and correctly seemed to generalize to the scenarios given. The derived gradients were correct (if a little too succiently explained for my taste). The experimental results themselves were strong (though I am not an expert in any of the areas so hard to say whether they picked the exact best things to compare against) and the additional quantitate analysis was a good decision since it shows there will be a consistent way to interpret results. I think the method seems to actually produce good results, has a good intuition behind it and could actually be useful which in my mind is much more important than excessively long proofs. There were a few questions about the experiment analysis that should have been addressed. In table 2 it’s odd for example that both P-EMB and AP-EMB both perform worse from K=50 to K=100. Since this measures purely the likelihood and does not include the prior theoretically increasing the size of K should only ever improve the score. In the worst case, letting the last 50 embedding and context vector values always be zero and letting the first 50 be the same as the optimal result for K=50 should give you a score as before. This seems to suggest that the SGD has some problems converging for higher dimensions so it would have been nice to have this addressed (e.g: was the convergence stopped due to time constraints). Generally mentioning the running times of EF-EMB methods versus the alternatives would have also been informative. One aspect I found confusing is that the link function introduced in equation 2 was generally assumed to be the identity function. However this runs into problems when the size of the context for each observation point is not constant (as was noted in the experimental analysis of the movie ratings). It seems like taking the dot product of the latent embedding and the weighted average of the context vectors is more generally the correct option.    Novelty/originality: 3  There is no new algorithm here but the idea of putting many different embeddings under the same sort of framework is new. The main novel contribution is noting that this could be done and esthablishing which key steps are required to do this. Moreover, each of the 4 exponential family versions of the embeddings for the difference scenarios are a novel contribution and appear to be empirically useful formulations. The experimental methods are fairly standard but seem fully appropriate for the problem of comparing old methods with EF-EMB.  Potential Impact: 4  I feel the authors have esthablished several key things about this framework. Firstly they have shown that it is fairly general through good use of 4 very different examples which seem well served by the method. Secondly, they have shown that the method is fairly easy to apply to a problem if you are familiar with the problem and exponential families. Once I review EF it was fairly easy to follow how they derived these new models and this ease of understanding makes it likely to use. Finally, they have shown that this is useful both in terms of the quantitate results but also in terms of giving us some examples of how this can be used to interpret results for exploratory reasons. Though this third part is perhaps the one I would like to see improved the most since they have not talked about how this fares for problem where the dimensionality of the embedding is very high (K was limited to just 100 in the examples) nor shown results for the more well studied problem of word embeddings. As previously mentioned there is no mention of how long it takes to run compared to other options which is also something that would help us understand how applicable the method might be.  Clarity and presentation: 3  Disclaimer, I last looked at exponential families 5 or so years ago so had to refresh myself on this several times so keep that in mind when reading this section. In my mind this was the weakest area of the paper and also the least balanced. I though the paper was very well motivated as we moved through sections 1 and 2: I was following the idea well, understood the idea of what needs to be defined to make an EF-EMB etc. Things starting getting a little harder to follow around 2.1. I think the authors presume a fair amount of familiarity with exponential families and would have benefitted from just stating things a little more explicitely. In example 1 of 2.1 for example there is discussion of a Gaussian with mean esthablished by the inner product but no mention of the standard deviation: it was only after a bit of reading that I saw this is replaced by lambda in the case of an exponential family prior-posterior case. The hardest aprt to follow was 2.2. Without referencing the canonical way to set up an exponential family conditional ( f(x| theta ) = h(x)exp(eta(theta). T(x) – A(theta) ) its not clear what terms like eta represent or to even find out what a(eta) is (which appears without warning in equation three first). Obviously its not a huge flaw but it does add a significant amount of effort to what was otherwise a pretty smooth read beforehand.  I was also a little confused by the use of “regularizer” in the objective function. From what I can understand this could more properly defined as the priors of the latent embeddings and context vectors and ensures that the objective function is seeking to maximize the entire posterior. Some of the equations’ derivations were a little terse. I can understand that in the interest of space not mentioning them in the main paper but it would have been nice to have a little discussion of what the prior is and how the chain rule is used in appendix A.1 for example. Again, nothing that can’t be worked out without a little extra research/calculations but neither is it something desirable. The experimental analysis was fairly easy to follow but I would have liked a little more detail on exactly what kind of factor analysis was being used for the first set of experiments as well as an explaination of why other parameter choices (such as higher values of k) were not explored.  Summary  Overall though I rather enjoyed reading this paper and think it is a great piece of work. It provides a framework for several ideas that is theoretically justified, useful and somewhat interpretable. The fact that it is fairly to simple to understand and does not rely on advanced mathematical machinery is yet another advantage. The generalizations seem generally good though there is certainly space for more investigation (e.g: can it be modified to admit cases where context data points can be weighted, e.g weighing the context word right next to the target word more than that five places to the right). The experimental work certainbly looks promising but I would like to see that get filled in with more analysis: especially on more computationally intensive scenarios.