Recent work has exhibited the surprising cross-lingual abilities of multilingual BERT (M-BERT) -surprising since it is trained without any cross-lingual objective and with no aligned data. In this work, we provide a comprehensive study of the contribution of different components in M-BERT to its cross-lingual ability. We study the impact of linguistic properties of the languages, the architecture of the model, and the learning objectives. The experimental study is done in the context of three typologically different languages -Spanish, Hindi, and Russian -and using two conceptually different NLP tasks, textual entailment and named entity recognition. Among our key conclusions is the fact that the lexical overlap between languages plays a negligible role in the cross-lingual success, while the depth of the network is an integral part of it.
What is the task?
Comprehensive study of the contribution of different components in Multilingual BERT to its cross-lingual ability.
What has been done before?
(Wu & Dredze, 2019) and (Pires et al., 2019) identified the cross-lingual success of the model and tried to understand it. However, both works treated M-BERT as a black box and compared M-BERT’s performance on different languages. This work, on the other hand, examines how B-BERT performs cross-lingually by probing its components, along multiple aspects. Some of the architectural conclusions have been observed earlier, if not investigated, in other contexts.
Authors claim that “Pires et al. (2019) hypothesizes that the cross-lingual ability of M-BERT arises because of the shared word-pieces between source and target languages.” is not entirely correct. Pires et al. (2019) did show M-BERT’s ability to transfer between languages that are written in different scripts, and thus have effectively zero lexical overlap. There were results (e.g. Figure 1) showing that while performance using EN-BERT depends directly on word piece overlap, M-BERT’s performance is largely independent of overlap, indicating that it learns multilingual representations deeper than simple vocabulary memorization.
Pires et al. (2019) also showed that structural similarity is crucial for cross-lingual transfer
What are the main contributions of the paper?
First comprehensive study of the contribution of different components in Multilingual BERT to its cross-lingual ability. Novel findings about the effect of network architecture, input representation and learning objective on cross lingual ability of M-BERT
Methodology that facilitates the analysis of similarities between languages and their impact on cross-lingual models by mapping English to a Fake-English language, that is identical in all aspects to English but shares no word-pieces with any target language.
What are the key dimensions studied/analyzed?
Different components/aspects of Multilingual BERT investigated:
(i) Linguistics properties and similarities of target and source languages  (has been studied in prior work)
(ii) Network Architecture (novel)
(iii) Input and Learning Objective (moderately novel)
What are the main results? Are they significant?
Lexical overlap between languages plays a negligible role in the cross-lingual success, while the depth of the network is an important part of it.  
Strengths 
Novel findings about the effect of network architecture, input representation and learning objective on cross lingual ability of M-BERT
Weaknesses
Pires et al. (2019) work has been misrepresented. (see above for more details)
Pires et al. (2019) did  study linguistics properties and similarities of target and source languages for Multilingual BERT and had similar findings as this work.
Questions
What kind of difference in the numbers is considered significant by the authors ? For example, according to them, an increase of more than 2 points in accuracy in Table 3 (e.g. 1 head vs. 6 heads) is considered insignificant. But a decrease of a point in accuracy in Table 5  (e.g. enfake-ru NSP vs. No-NSP) is significant.