MULTILINGUAL NEURAL MACHINE TRANSLATION WITH SOFT DECOUPLED ENCODING
Multilingual training of neural machine translation (NMT) systems has led to impressive accuracy improvements on low-resource languages. However, there are still significant challenges in efficiently learning word representations in the face of paucity of data. In this paper, we propose Soft Decoupled Encoding (SDE), a multilingual lexicon encoding framework specifically designed to share lexicallevel information intelligently without requiring heuristic preprocessing such as pre-segmenting the data. SDE represents a word by its spelling through a character encoding, and its semantic meaning through a latent embedding space shared by all languages. Experiments on a standard dataset of four low-resource languages show consistent improvements over strong multilingual NMT baselines, with gains of up to 2 BLEU on one of the tested languages, achieving the new state-of-the-art on all four language pairs 1 .
This paper focuses on the problem of word representations in multilingual NMT system. The idea of multilingual NMT is to share data among multiple language pairs. Crucially this requires some way to tie the parameters of words from different languages, and one popular method is to share subword units among languages. The problem is that subword units in different languages may not be semantically equivalent, and many semantically-equivalent concepts are not represented by the same subwords. This paper proposes an alternative way to share word representation, in particular by proposing a common set of "semantic" concept vectors across languages which are then folded into the word representations via attention. 
The problem is well-motivated and the proposed solution is reasonable. Previous works such as (Gu et. al. 2018) have been motivated in a similar fashion, and the proposed solution seems to outperform it on the TED dataset of Qi et. al. 2018. 
The experiments are informative. The main open questions I have are:
(a) Varying the latent embedding size. It seems like only 10,000 is tried. Since this is the main contribution of the work, it will be desirable to see results for different sizes. Is the method sensitive to this hyperparameter? Also suggestions on how to pick the right number based on vocabulary size, sentence size, or other language/corpus characteristics will be helpful. 
(b) What do the latent embeddings look like? Intuitively will they be very different from those from Gu et. al. 2018 because you are using words rather than subwords as the lexical unit? 
(c) The explanation for why your model outperforms Gu et. al. 2018 seems insufficient -- it would be helpful to provide more empirical evidence in the ablation studies in order really understand why your method, which is similar to some extent, is so much better. 
The paper is generally clear. Here are few suggestions for improvement:
- Table 1: Please explain lex unit, embedding, encoding in detail. For example, it is not clear what is joint-Lookup vs. pretrain-Lookup. It can be inferred if one knows the previous works, but to be self-contained, I would recommend moving this table and section to Related Works and explaining the differences more exactly.
- Sec 4.2: Explain the motivation for examining the three different lexical units. 
- Table 3: "Model = Lookup (ours)" was confusing. Do you mean "our implementation of Neubig & Hu 2018? Or ours=SDE? I think the former?
- Are the word representions in Eq 4 defined for each word type or word token? In other words, for the same word "puppy" in two different sentences in the training data, do they have the same attention and thus the same e_SDE(w)? You do not have different attentions depending on the sentence, correct? I think so, but please clarify. (Actually, Figure 2 has a LSTM which implies a sentential context, so this was what caused the potential confusion). 
- There are some inconsistencies in the terms: e.g. latent semantic embedding vs latent word embedding. Lexical embedding vs Character embedding. This makes it a bit harder to line up Sec 4.4 results with Sec 3.2 methods. 
- Minor spelling mistakes. e.g. dependant -> dependent. Please double-check for others.