In this paper, we introduce MAMUS for constructing multilingual sparse word representations. Our algorithm operates by determining a shared set of semantic units which get reutilized across languages, providing it a competitive edge both in terms of speed and evaluation performance. We demonstrate that our proposed algorithm behaves competitively to strong baselines through a series of rigorous experiments performed towards downstream applications spanning over dependency parsing, document classification and natural language inference. Additionally, our experiments relying on the QVEC-CCA evaluation score suggests that the proposed sparse word representations convey an increased interpretability as opposed to alternative approaches. Finally, we are releasing our multilingual sparse word representations for the 27 typologically diverse set of languages that we conducted our various experiments on.
This paper describes a method to build sparse multilingual word vectors that is designed to scale easily to many languages. The key idea is to pick one language to be the source language, and then to build word embeddings and then a sparse dictionary + sparse coefficients  for that source language monolingually. Other target languages then first align their embedding to the source using a seed list of translations and standard techniques, and then determine their sparse coefficients based on the fixed source sparse dictionary. This latter process is a convex optimization, which improves efficiency and stability. The method is tested with an established correlation-based intrinsic metric (QVEC-CCA) as well as by using the multilingual embeddings to project systems for cross-lingual document classification, dependency parsing and natural language inference.
The core idea of this paper is substantially simpler than the method it compares to (BiSparse), which jointly optimizes dictionaries, coefficients and cross-lingual coefficient alignment. So, the question that immediately comes to mind for me is, how much accuracy am I giving up for improved scalability to multiple languages? This could be easily tested for the two language case, where BiSparse could be directly compared without modification to their proposed method. Instead, BiSparse is modified to fit scale to the multilingual setting (becoming MultiSparse), but since it shares the constraint that the source dictionary and coefficients are fixed, it has already lost a lot of the power of joint optimization. I think the paper would be stronger with a two-language experiment where we would expect the proposed method to lose to BiSparse, but we could begin to understand what has been given up for scalability.
I also wonder how relevant bilingual word embeddings are in a world of multilingual BERT and similar approaches. It would be interesting to know how cross-lingual embedding-in-context methods would do on the extrinsic evaluations in this paper, though I also acknowledge that this could be considered beyond the scope of the paper.
Otherwise, this is a fine paper. It is well written and easy to follow. The experiments look sane, and the inclusion of both intrinsic and extrinsic tasks makes them fairly convincing. I have only a few remaining nitpicks:
(1) As someone relatively unfamiliar with multilingual (as opposed to bilingual) word embedding research, it wasn’t clear to me how the experiments described here tested the multilinguality (as opposed to bilinguality) of the embeddings. It would be nice to provide an explanation for why (beyond the obvious efficiency gains) one couldn’t just do the necessary language pairs with bilingual methods for these experiments. And if one could perform the tests with bilingual methods, they should be included as baselines. 
(2) The discussion of MultiSpare hyper-parameter tuning appears in the Monolingual Experiments section, leading me to wonder what target languages were used for this tuning process.
(3) In the second-last paragraph of 3.2.2, there is a sentence fragment that ends in “nonetheless their multiCluster and multiCCA embedding spaces contain no embeddings for”
(4) The last paragraph before the Conclusion also feels like a fragment, or like two sentences have been spliced together: “We have detailed the differences to Upadhyay et al. (2018) extends the previous work by incorporating dependency relations into sparse coding.”
There are also several places where periods seem to have been left out (such as immediately before the above sentence).