Word embeddings are known to boost performance of many NLP tasks such as text classification, meanwhile they can be enhanced by labels at the document level to capture nuanced meaning such as sentiment and topic. Can one combine these two research directions to benefit from both? In this paper, we propose to jointly train a text classifier with a label-enhanced and domain-aware word embedding model, using an unlabeled corpus and only a few labeled data from non-target domains. The embeddings are trained on the unlabed corpus and enhanced by pseudo labels coming from the classifier, and at the same time are used by the classifier as input and training signals. We formalize this symbiotic cycle in a variational Bayes framework, and show that our method improves both the embeddings and the text classifier, outperforming state-of-the-art domain adaptation and semisupervised learning techniques. We conduct detailed ablative tests to reveal gains from important components of our approach. The source code and experiment data will be publicly released.
The authors propose a semi-supervised learning techhnique involving jointly tuning word embeddings and a classifier. The idea is to rapidly adapt models to new domains with minimal (actually, zero) supervision by exploiting such embeddings.
In effect, this is an approach to "disentangling" sentiment from domain, via a variational objective and semi-supervision via the embedding parameters. This is a nice idea as disentanglement enables transfer (or should). The experiments are well-executed (albeit on a limited set of classification tasks). Still the comparisons are relatively exhaustive, and the authors have done a nice job of providing ablations. I found Table 4 and Figure 2 particularly nice. 
- The authors should speak to the generalizability of this approach beyond sentiment tasks, as presently it seems largely constrained to this domain. This would of course hinder the potential utility/impact of the work.
- Due to the very concise description of baselines, it was not obvious to me if these were also taking a semi-supervised approach? I *think* the pivot-based model is. But then how was CNN+ELMo trained exactly? With zero target sentiment labels, I presume? If so, an obvious baseline would be to "pseudo-label" instances in the target domain first, and then use these predictions as a target to fine-tune the CNN (or whatever), back-propping through to the embeddings. Was this done? In general more details on the baselines and training setups should be included, even if only in the Appendix.
- The strategy, I think, is to learn a prediction model p_\phi(y|z, D) where \phi are model parameters. Then this is applied to instances in the target dataset to infer latent z, and then use these inferred labels to fine-tune word embeddings. Specifically, sentiment label information is pushed into embeddings via a modified CBOW objective that effectively shifts the meaning (as codified in the distributed representation) via task specific sentiment. One thing here that confused me is that this would seem to perform this affine transformation to all words, but only certain of these will exhibit domain specific variation with respect to sentiment. Can the authors speak to this? 
- I found Eq 7 counterintuitive, since it treats sentiment and domain independently, but the authors explicitly noted above that this is not the case, i.e., words will depend jointly on sentiment + domain. I actually think p(w_i|y) also depends on the domain via M (implicitly), but perhaps this should be made explicit (or perhaps I am misunderstanding something!)
- In 2.1, the authors assume a prior p(z|c) and then say that `'naively' marginalizing over this performs poorly. But I not think the particular prior distribution used was not discussed here. Could they elaborate? 
- I think i indices are needed on the w's in Eq 6?