Word embedding is a useful approach to capture co-occurrence structures in a large corpus of text. In addition to the text data itself, we often have additional covariates associated with individual documents in the corpus-e.g. the demographic of the author, time and venue of publication, etc.-and we would like the embedding to naturally capture the information of the covariates. In this paper, we propose a new tensor decomposition model for word embeddings with covariates. Our model jointly learns a base embedding for all the words as well as a weighted diagonal transformation to model how each covariate modifies the base embedding. To obtain the specific embedding for a particular author or venue, for example, we can then simply multiply the base embedding by the transformation matrix associated with that time or venue. The main advantages of our approach is data efficiency and interpretability of the covariate transformation matrix. Our experiments demonstrate that our joint model learns substantially better embeddings conditioned on each covariate compared to the standard approach of learning a separate embedding for each covariate using only the relevant subset of data. Furthermore, our model encourages the embeddings to be "topic-aligned" in the sense that the dimensions have specific independent meanings. This allows our covariate-specific embeddings to be compared by topic, enabling downstream differential analysis. We empirically evaluate the benefits of our algorithm on several datasets, and demonstrate how it can be used to address many natural questions about the effects of covariates.
The authors present a method for learning word embeddings from related groups of data. The model is based on tensor factorization which extends GloVe to higher order co-ocurrence tensors, where the co-ocurrence is of words within subgroups of the text data. These two papers need to be cited:
Rudolph et al., NIPS 2017, "Sturctured Embedding Models for Grouped Data": This paper also presents a method for learning embeddings specific for subgroups of the data, but based on hierarchical modeling. An experimental comparison is needed.
Cotterell et al., EACL 2017 "Explaining and Generalizing Skip-Gram through Exponential Family Principal Component Analysis": This paper also derives a tensor factorization based approach for learning word embeddings for different covariates. Here the covariates are morphological tags such as part-of-speech tags of the words.
Due to these two citations, the novelty of both the problem set-up of learning different embeddings for each covariate and the novelty of the tensor factorization based model are limited.
The writing is ok. I appreciated the set-up of the introduction with the two questions. However, the questions themselves could have been formulated differently: 
Q1: the way Q1 is formulated makes it sound like the covariates could be both discrete and continuous while the method presented later in the paper is only for discrete covariates (i.e. group structure of the data).
Q2: The authors mention topic alignment without specifying what the topics are aligned to. It would be clearer if they stated explicitly that the alignment is between covariate-specific embeddings. It is also distracting that they call the embedding dimensions topics.
Also, why highlight the problem of authorship attribution of Shakespear's work in the introduction, if that problem is not addressed later on?
In the model section, the paragraphs "notation" and "objective function and discussion" are clear.  I also liked the idea of having the section "A geometric view of embeddings and tensor decomposition", but that section needs to be improved. For example, the authors describe RandWalk (Arora et al. 2016) but how their work falls into that framework is unclear.
In the third paragraph, starting with "Therefore we consider a natural extension of this model, ..." it is unclear which model the authors are referring to. (RandWalk or their tensor factorization?).
What are the context vectors in Figure 1?  I am guessing the random walk transitions are the ellipsoids? How are they to be interpreted? 
In the last paragraph, beginning with "Note that this is essentially saying...", I don't agree with the argument that the "base embeddings" decompose into independent topics. The dimensions of the base embeddings are some kind of latent attributes and each individual dimension could be used by the model to capture a variety of attributes. There is nothing that prevents the model from using multiple dimensions to capture related structure of the data. Also, the qualitative results in Table 3 do not convince me that the embedding dimensions represent topics. For example "horses" has highest value in embedding dimension 99. It's nearest neighbours in the embedding space (i.e. semantically similar words) will also have high values in coordinate 99. Hence, the apparent semantic coherence in what the authors call "topics".
The authors present multiple qualitative and quantitative evaluations. The clustering by weight (4.1.) is nice and convincing that the model learns something useful. 4.2, the only quantitative analysis was missing some details. Please give references for the evaluation metrics used, for proper credit and so people can look up these tasks. Also, comparison needed to fitting GloVe on the entire corpus (without covariates) and existing methods Rudolph et al. 2017 and Cotterell et al. 2017.   
Section 5.2 was nice and so was 5.3. However, for the covariate specific analogies (5.3.) the authors could also analyze word similarities without the analogy component and probably see similar qualitative results. Specifically, they could analyze for a set of query words, what the most similar words are in the embeddings obtained from different subsections of the data.
PROS:
+ nice tensor factorization model for learning word embeddings specific to discrete covariates.
+ the tensor factorization set-up ensures that the embedding dimensions are aligned 
+ clustering by weights (4.1) is useful and seems coherent
+ covariate-specific analogies are a creative analysis
CONS:
- problem set-up not novel and existing approach not cited (experimental comparison needed)
- interpretation of embedding dimensions as topics not convincing
- connection to Rand-Walk (Aurora 2016) not stated precisely enough
- quantitative results (Table 1) too little detail:
        * why is this metric appropriate?
        * comparison to GloVe on the entire corpus (not covariate specific)
        * no reference for the metrics used (AP, BLESS, etc.?)
- covariate specific analogies presented confusingly and similar but simpler analysis might be possible by looking at variance in neighbours v_b and v_d without involving v_a and v_c (i.e. don't talk about analogies but about similarities)