Structured Embedding Models for Grouped Data
Word embeddings are a powerful approach for analyzing language, and exponential family embeddings (EFE) extend them to other types of data. Here we develop structured exponential family embeddings (S-EFE), a method for discovering embeddings that vary across related groups of data. We study how the word usage of U.S. Congressional speeches varies across states and party affiliation, how words are used differently across sections of the ArXiv, and how the copurchase patterns of groceries can vary across seasons. Key to the success of our method is that the groups share statistical information. We develop two sharing strategies: hierarchical modeling and amortization. We demonstrate the benefits of this approach in empirical studies of speeches, abstracts, and shopping baskets. We show how S-EFE enables group-specific interpretation of word usage, and outperforms EFE in predicting held-out data.
This paper presents a word embedding model for grouped data. It extends EFE to learn group-specific embedding vectors, while sharing the same context vector. To handle groups limited data, the authors propose two methods (hierarchical and amortization) to derive group-specific embedding vectors from a shared one. The paper is clearly written, but the novelty is a bit limited since it is an incremental work beyond EFE.
1. From Table 2, there is no clear winner among the three proposed models (hierarchical, amortiz+feedf, and amortiz+resnet), and the performance differences are subtle especially on the shopping data. If one would like to use S-EFE, do you have any practical guidance on choosing the right model? I guess we should prefer amortiz+resnet to amortiz+feedf, since amortiz+resnet always outperforms amortiz+feedf. Line 276 mentions that hierarchical S-EFE works better when there are more groups. Why?
2. Why Separate EFE performs worse than Global EFE?
3. The authors proposed hierarchical and amortization methods for the reasons in lines 128-131. It is interesting to see how S-EFE performs w.r.t. various data sizes. From this experiment, we might understand if S-EFE is going to surpass hierarchical/amortiz S-EFE, provided more and more data. If yes, would it be better to promote S-EFE as the leading method? If not, why?
4. I understand that pseudo log likelihood is a standard metric for evaluating embedding methods. However, it would be great to see how embedding helps in standard language understanding tasks, like text classification, machine translation, etc.
5. lines 141 and 143: Eq. 9 to Eq. 4