Modern natural language processing and understanding applications have enjoyed a great boost utilizing neural networks models. However, this is not the case for most languages especially low-resource ones with insufficient annotated training data. Cross-lingual transfer learning methods improve the performance on a lowresource target language by leveraging labeled data from other (source) languages, typically with the help of cross-lingual resources such as parallel corpora. In this work, we propose a zero-resource multilingual transfer learning model 1 that can utilize training data in multiple source languages, while not requiring target language training data nor cross-lingual supervision. Unlike most existing methods that only rely on language-invariant features for cross-lingual transfer, our approach utilizes both language-invariant and language-specific features in a coherent way. Our model leverages adversarial networks to learn language-invariant features and mixture-of-experts models to dynamically exploit the relation between the target language and each individual source language. This enables our model to learn effectively what to share between various languages in the multilingual setup. It results in significant performance gains over prior art, as shown in an extensive set of experiments over multiple text classification and sequence tagging tasks including a large-scale real-world industry dataset.
This paper describes a model for cross lingual transfer with no target language information. This is a well written paper that makes a number of contributions:
1. It provides an interesting discussion of transfer form multiple source languages into a target language. This is a timely problem and the paper points out that adversarial networks may be too limiting in this setup.
2. It provides a modeling approach that deals with the limitations of adversarial networks as mentioned in (1).
3. It demonstrates the value of the proposed approach through an extensive experimental setup.
At the same time, I see two major limitations to the paper:
1. While the proposed approach is valid, it is not very original, at least in my subjective eyes. The authors integrate a classifier that combines the private, language-specific features so that not only features that are shared between all the involved languages can be used in the classification process. While this is a reasonable idea that works well in practice, IMO it is quite straight forward and builds on ideas that have been recently been proposed in many other works.
2. The authors claim that: "To our best knowledge, this work is the first to propose
an unsupervised CLTL framework without depending on any cross-lingual resource"
This is, unfortunately, not true. I refer the authors to the paper:
Deep Pivot-Based Modeling for Cross-language Cross-domain Transfer with Minimal Guidance. Yftah Ziser and Roi Reichart. EMNLP 2018.
In their lazy setup, the EMNLP authors do exactly that. They address the more complicated cross-language, cross-domain setup, but their model can be easily employed within a single domain. Their experiments even use the multilingual sentiment dataset used in the current paper. The model in the EMNLP paper shows to outperform adversarial networks, so it can be competitive here as well.