CBOW IS NOT ALL YOU NEED: COMBINING CBOW WITH THE COMPOSITIONAL MATRIX SPACE MODEL
Continuous Bag of Words (CBOW) is a powerful text embedding method. Due to its strong capabilities to encode word content, CBOW embeddings perform well on a wide range of downstream tasks while being efficient to compute. However, CBOW is not capable of capturing the word order. The reason is that the computation of CBOW's word embeddings is commutative, i.e., embeddings of XYZ and ZYX are the same. In order to address this shortcoming, we propose a learning algorithm for the Continuous Matrix Space Model (Rudolph & Giesbrecht, 2010), which we call Continual Multiplication of Words (CMOW). Our algorithm is an adaptation of word2vec (Mikolov et al., 2013a), so that it can be trained on large quantities of unlabeled text. We empirically show that CMOW better captures linguistic properties, but it is inferior to CBOW in memorizing word content. Motivated by these findings, we propose a hybrid model that combines the strengths of CBOW and CMOW. Our results show that the hybrid CBOW-CMOW-model retains CBOW's strong ability to memorize word content while at the same time substantially improving its ability to encode other linguistic information by 8%. As a result, the hybrid also performs better on 8 out of 11 supervised downstream tasks with an average improvement of 1.2%.
The authors propose CMOW, an extension of the CBOW model that allows the model to capture word order. Instead of each word being represented as a vector, words are represented by matrices. They extend the CBOW objective to take into account word order by replacing the averaging of vectors to create the context with matrix multiplication (a non-commutative operation). This is the first time this model has been applied in a large scale unsupervised setting. They are able to do this using their objective and an initialization strategy where the matrix embeddings are set to the identity matrix with some Gaussian noise added.
The results of this paper are its main weakness. I did enjoy reading the paper, and it is nice to see some results using matrices as embeddings and matrix multiplication as a compositional function. They include a nice analysis of how word order is captured by these CMOW embeddings while CBOW embeddings capture the word content, but it doesn't seem to make much of a difference on the downstream tasks where CBOW is better than CMOW and close to the performance of the hybrid combination of CBOW and CMOW.
I think it's clear that their model is able to capture word information to some extent, but other models  (RNNs etc.) can do this as well, that admittedly are more expensive, but also have better performance on downstream tasks. I think a stronger motivation for their method besides an analysis of some phenomena it captures and a slight improvement on some downstream tasks when combined with CBOW is needed though for acceptance. Could it be used in other settings besides these downstream transfer tasks?
PROS:
- introduced an efficient and stable approach for training CMSM models
- Show that their model CMOW is able to capture word order information
- Show that CMOW compliments CBOW and a hybrid model leads to improved results on downstream tasks. 
CONS
- The results on the hybrid model are only slightly better than CBOW. CMOW alone is mostly worse than CBOW.