text:
\section{Introduction}

Recent pre-trained multilingual transformer models, such as XLM(-R) \cite{conneau2019cross,xlm-r}, mT5 \cite{mt5} and others \cite{liu2020multilingual,lewis2020pre} have been shown to be  successful in NLP tasks for several non-English languages \citep{parsinlu,choi2021analyzing},
as well as in multilingual benchmarks \citep{bert,xlm-r,mt5,artetxe-etal-2020-cross}. A particular appeal is that they can be used for \emph{cross-lingual} and \emph{zero-shot transfer}. That is, after pre-training on a raw, unaligned corpus consisting of text from many languages,  models can be subsequently fine-tuned on a particular task in a resource-rich language (e.g., English) and directly applied to the same task in other languages without requiring any additional language-specific training.

Given this recent progress, a natural question arises: does it make sense to invest in large-scale task-specific dataset construction for low-resourced languages, or does cross-lingual transfer alone suffice for many languages and tasks? A closely related question is: how well do multilingual models transfer across specific linguistic and language-specific phenomena? While there has been much recent work on probing multilingual models \cite{wu2019mbert,pires2019mbert,karthikeyan2019mbert}, \emph{inter alia}, a particular limitation is that most studies rely on automatically translated resources such as XNLI \cite{xnli} and XQuAD \cite{artetxe-etal-2020-cross}, which makes it difficult to discern the particular linguistic knowledge that  is  being  transferred  and  the  role of large-scale, expert annotated monolingual datasets when building task- and language-specific models.

In this paper, we investigate the cross-lingual transfer abilities of XLM-R \cite{xlm-r} for Chinese natural language inference (NLI). Our focus on Chinese NLI is motivated by the recent release of the first large-scale, human-annotated Chinese NLI dataset OCNLI (\textbf{O}\emph{riginal} \textbf{C}\emph{hinese} \textbf{NLI}) \cite{ocnli}\footnote{To our knowledge, OCNLI is currently the largest non-English NLI dataset that was annotated in the style of English MNLI without any translation.}, which we use to directly investigate the role of high-quality task-specific data vs. English-based cross-lingual transfer. To better understand linguistic transfer, and help benchmark recent SOTA Chinese NLI models, we created 4 categories of challenge/adversarial tasks (totaling 17 new datasets) for Chinese that build on  several  well-established resources for English and the literature on model probing (see \citet{poliak2020survey}). Our new resources, which are summarized in Table~\ref{tab:sum:4sets}, include: a new set of diagnostic tests in the style of the SuperGLUE ~\citep{superglue} and CLUE \cite{clue} diagnostics; Chinese versions of the HANS dataset \cite{hans} and NLI stress-tests \cite{naik2018stress}, as well as a collection of the basic reasoning and logic \emph{semantic probes} for Chinese based on \citet{probing2020}.

\begin{table}[t]
\centering
\footnotesize
\scalebox{.85}{
\begin{tabular}{lll}\toprule
 & category & n  \\\midrule
\multirow{2}{*}{\rotatebox[origin=c]{90}{\parbox{.8cm}{Chinese\\HANS}}} & \parbox{2cm}{Lexical overlap} & 1,428   \\
 & Subsequence & 513   \\\midrule
\multirow{5}{*}{\rotatebox[origin=c]{90}{stress tests}}
& Distraction: 2 categories & 8,000   \\
 & Antonym & 3,000   \\
 & Synonym & 2,000   \\
 & Spelling & 11,676  \\
 & Numerical reasoning & 8,613   \\\midrule
\multirow{7}{*}{\rotatebox[origin=c]{90}{diagnostics}} & CLUE~\cite{clue} & 514   \\
 & CLUE expansion (ours) & 796   \\
 & World knowledge (ours) & 38   \\
 & Classifier (ours) & 139   \\
 & Chengyu/idioms (ours) & 251   \\
 & Pro-drop (ours) & 198   \\
 & Non-core arguments (ours) & 186 \\\midrule
\multirow{6}{*}{\rotatebox[origin=c]{90}{\parbox{1.5cm}{semantic\\probing}}} & Negation & 1,002   \\
 & Boolean & 1,002   \\
 & Quantifier & 1,002   \\
 & Counting & 1,002   \\
 & Conditional & 1,002   \\
 & Comparative & 1,002\\\midrule
sum & & 43,364\\\bottomrule
\end{tabular}
}
\caption{Summary statistics of the four evaluation sets. \label{tab:sum:4sets}}
\end{table}

Our results are largely positive: We find that cross-lingual models trained exclusively on English NLI do transfer relatively well across our new Chinese tasks (e.g., in 3/4 of the challenge categories shown in Table~\ref{tab:sum:4sets}, they perform overall as well or better than the best monolingual Chinese models without additional specialized training on Chinese data, and have competitive performance on OCNLI).  A particularly striking result is that such models even perform well on 3/5 uniquely Chinese linguistic phenomena such as \emph{idioms, pro drop}, providing evidence that many language-specific  phenomena do indeed transfer. These results, however, come with important caveats: on several phenomena we find that models continue to struggle and are far outpaced by conservative estimates of human performance (e.g., our best model on Chinese HANS remains $\sim$19\% behind human performance), highlighting the need for more language-specific diagnostics tests. Also, fine-tuning models on mixtures of English NLI data and high-quality monolingual data (OCNLI) consistently performs the best, whereas mixing with automatically translated datasets (XNLI-zh) can greatly hinder model performance. This last result shows that high-quality monolingual datasets still play an important role when building cross-lingual models, however, the particular type of monolingual dataset that is needed can vary and is best informed by targeted behavioral testing of the type we pursue here.

\section{Related Work}

There has been a lot of work on trying to understand  multilingual transformers \cite{wu2019mbert,pires2019mbert}, which has focused on either examining the representation of different layers in the transformer architecture or the lexical overlap between languages. \citet{karthikeyan2019mbert} investigate the role of network depth and number of attention heads, as well as
syntactic/word-order similarity on the cross-lingual transfer performance.
In addition to studies cited at the outset, positive results of cross-lingual transfer across a wide range of languages are reported in \citet{wu2020all,nozza2020mask}, with a focus on transfer across specific tasks such as POS tagging, NER; in contrast, we focus on different categories of linguistic transfer, which has received less attention, as well as the role of monolingual data for transfer in NLI.

Studies into the linguistic abilities and robustness of current NLI models have proliferated in recent years, partly owing to the discovery of systematic biases, or \emph{annotation artifacts} \cite{gururangan2018annotation,poliak2018hypothesis}, in benchmark NLI datasets such as SNLI \cite{snli} and MNLI \cite{mnli}. This has been coupled with the development of new \emph{adversarial tests} such as  \emph{HANS} \cite{hans} and the NLI \emph{stress-tests} \cite{naik2018stress}, as well as several new linguistic \emph{challenge datasets} \cite{breaknli,probing2020,geiger-etal-2020-neural,yanaka2019help,saha2020conjnli,goodwin2020ystematicity}, \emph{inter alia}, that focus on a wide range of linguistic and reasoning phenomena. All of this work focuses exclusively on English, whereas we focus on constructing analogous probing datasets tailored to Chinese to help advance research on Chinese NLI and cross-lingual transfer.

There has been a surge in the development
of NLI resources for languages other than English. Such resources are often created in the following two ways:
(1) from \emph{scratch}, in the style of MNLI \cite{mnli}, where annotators are used to produce hypotheses and inference labels based on a provided set of premises, as pursued for Chinese OCNLI \cite{ocnli}, or SciTail \cite{scitail}, where sentences are paired automatically and labeled by annotators \cite{amirkhani2020farstail,hayashibe2020japanese}. (2) Through \emph{automatic} \cite{xnli,turkish-nli,assin2} or \emph{manual} \cite{sick-dutch} translation from existing English datasets. Studies on cross-lingual transfer for NLI have largely focused on XNLI \cite{xnli}, which we show has limited utility for Chinese NLI transfer.

\section{Dataset creation}

In this section, we describe the details of the 4 types of challenge datasets we constructed for Chinese to study cross-lingual transfer (see details in Table~\ref{tab:sum:4sets}). They fit into two general categories: \textbf{Adversarial datasets} (Section~\ref{sec:ad}) built largely from patterns in OCNLI~\cite{ocnli} and XNLI~\cite{xnli} and \textbf{Probing/diagnostic datasets} (Section~\ref{sec:prob}), which are \orange{built from scratch in a parallel fashion to existing datasets in English}.

While we aim to mimic the annotation protocols pursued in the original English studies,  we place the additional methodological constraint that each new dataset is vetted, either through human annotation using a disjoint set of Chinese linguists, or through internal mediation among local Chinese experts; details are provided below.

\subsection{Adversarial dataset}
\label{sec:ad}

Examples from the 7 adversarial tests we created are illustrated in Table~\ref{tab:ex:hans:stress}.\footnote{A more detailed description of
the data creation process
can be found in Appendix~\ref{sec:appendix:ex}.} Chinese HANS is built from patterns extracted in the large-scale Chinese NLI dataset OCNLI \cite{ocnli}, whereas the \textbf{Distraction}, \textbf{Antonym}, \textbf{Synonym} and \textbf{Spelling} subsets are built from an equal mixture of OCNLI and XNLI-zh \cite{xnli} data; in the latter case, such a difference allows us to fairly compare the effect of training on expert-annotated (i.e., OCNLI) vs. automatically translated data (i.e., XNLI-zh) as detailed in Section~\ref{sec:exp}.

\paragraph{Chinese HANS}

\citet{hans} discovered systematic biases/heuristics in the MNLI dataset,
which they named ``lexical/subsequence/constituent'' overlap.
``Lexical overlap'' is defined to be the pairs where the vocabulary of the hypothesis
is a subset of the vocabulary of the premise.
For example, ``\textit{The boss is meeting the client.}'' and ``\textit{The client is meeting the boss.}'',
which has an entailment relation.
However, lexical overlap does not necessarily mean the premise
will entail the hypothesis, e.g., ``\textit{The judge was paid by the actor.}'' does not entail ``\textit{The actor was paid by the judge.}'' (examples from \citet{hans}). Thus a model relying on the heuristic will fail catastrophically in the second case.

Inspired by the English HANS, we examine whether OCNLI also possesses
such biases, as it has a similar annotation procedure as MNLI.
We follow the design of the original HANS experiments, and adapt
their scripts\footnote{\url{https://github.com/tommccoy1/hans}} to extract examples in OCNLI that satisfy the two heuristics. We find a heavy bias towards ``entailment'', where 79.5\% of such
examples are ``entailment'', similar to MNLI.
To construct a Chinese HANS, we first look into syntactic structures of the examples having the two heuristics. Then we write 29 templates for the \textit{lexical overlap} heuristic and 11 templates for  \textit{subsequence overlap}.\footnote{For details of the templates, see Appendix~\ref{sec:app:hans:templates}.
}
Using the templates and a vocabulary
of 263 words, we generated
1,941 NLI pairs. See Table~\ref{tab:ex:hans:stress} for examples and Appendix~\ref{sec:appendix:ex} for details.

\begin{CJK*}{UTF8}{gbsn}
\begin{table*}[t]
\scriptsize
\centering
\scalebox{.95}{
\begin{tabular}{p{.2cm}p{1cm}p{.3cm}p{7cm}p{4.5cm}p{.3cm}}\toprule
 & category & n & premise & hypothesis & label \\\midrule
\multirow{2}{*}{\rotatebox[origin=c]{90}{\parbox{.8cm}{Chinese \\ HANS}}} & Lexical overlap & 1428 & 我们把银行职员留在电影院了。  We left the bank clerk in the cinema. &银行职员把我们留在电影院了。  The bank clerk left us in the cinema. & C \\
 & Subsequence & 513 & 谁说\underline{律师都是穿西装的}。  Who told you that \ul{all lawyers wear suits}. & \underline{律师都是穿西装的}。   \ul{All lawyers wear suits}. & C \\\midrule
\multirow{6}{*}{\rotatebox[origin=c]{90}{\parbox{3cm}{stress tests}}} & Distraction \newline (add to\newline premise) & 4000 & 国有企业改革的思路和方针政策已经明确,而且刚\underline{做完手术出院的病人不应剧烈运动}。 The policy of the reform of state-owned enterprises is now clear, \ul{and patients who just had surgery shouldn't have intense exercise}. & 根本不存在国有企业。   The state-owned enterprises don't exist. & C \\
 & Distraction \newline (add to\newline hypothesis) & 4000 & 这时李家院子挤满了参观的人。  During this time, the Li family's backyard is full of people who came to visit. & 这地方有个姓李的人家, \underline{而且真的不是假的}。 There is a Li family here, and \ul{true is not false}. & E \\
 & Antonym & 3000 & 一些地方财政收支矛盾较大。    The disagreement about local revenue is relatively big. & 一些地方财政收支矛盾较小。  The disagreement about local revenue is relatively small. & C \\
 & Synonym & 2000 & 海部组阁困难说明了什么。  What can you tell from the \underline{difficulties} from Kaifu's attempt to set up a cabinet? & 海部组阁艰难说明了什么。  What can you tell from the \underline{hardships} from Kaifu's attempt to set up a cabinet? & E \\
 & Spelling & 2980 & 身上裹一件工厂发的棉大衣,手插在袖筒里。 (Someone is) wrapped up in a big cotton coat the factory gave with hands in the sleeves & 身上\underline{质}少一件衣服。 There's at \underline{least} [typo] one coat on the body. & E \\
 & Numerical\newline reasoning & 8613 & 小红每分钟打不到510个字。 Xiaohong types fewer than 510 words per min. & 小红每分钟打110个字。 Xiaohong types 110 words per min. & N \\\bottomrule
 \end{tabular}}
\caption{Example NLI pairs in Chinese HANS and stress tests with translations. \label{tab:ex:hans:stress}}
\end{table*}

\paragraph{Distraction}
We add distractions to the premise
or hypothesis, similar to the ``length
mismatch'' and ``word overlap'' conditions
in the NLI \emph{stress tests} of  \citet{naik2018stress}. The distractions are either tautologies (``true is not false'') or a true statement from our world knowledge (``Finnland is not a permanent  member of the UN security council''), which should
not influence the inference label.
We control whether the distraction
contain a negation or not, and thus create
four conditions: \textit{premise-negation}, \textit{premise-no-negation}, \textit{hypothesis-negation}, and \textit{hypothesis-no-negation}. See Table~\ref{tab:ex:hans:stress} for examples.

\paragraph{Antonym} We replace a word in the premise with its antonym to form a contradiction. To ensure the quality
of the resulting NLI pairs,
we manually examine the initially generated data and decided to only
replace nouns and adjectives, as
they are more likely to produce
real contradictions.

\paragraph{Synonym} We replace a word in the premise with its synonym to form an
entailment.

\paragraph{Spelling} We replace one random character in the hypotheses with its homonym (character with the same \textit{pinyin} pronunciation ignoring tones) as this is one of the most common types of misspellings in Chinese.

\paragraph{Numerical reasoning} We create a probing set for numerical reasoning,
following simple heuristics such as the following.
When the premise is \textit{Mary types x words per minute}, the entailed hypothesis can be: \textit{Mary types less than y words per minute}, where \textit{x} $<$ \textit{y}.
A contradictory hypothesis:
\textit{Mary types y words per minute}, where \textit{x} $>$ \textit{y} or \textit{x} $<$ \textit{y}.
Then a neutral pair can be produced by reversing the premise and hypothesis of the above entailment pair.
4 heuristic rules (with 6 words for quantification) are used and the seed sentences are
extracted from Ape210k \citep{ape210k}, a dataset of Chinese elementary-school math problems.
The resulting data contains 8,613 NLI pairs.

For \textbf{quality control} and to compute human performance, we randomly
sampled 50 examples from all subsets
and asked 5  Chinese speakers to verify. Our goal is to mimic the human annotation protocol from \citet{glue-human}, which gives us a \emph{conservative} estimate of human performance given that our annotators received very little instructions. Their majority vote agrees with the gold label
90.0\% of the time, which suggests that our data is of high quality and allows us to later compare against model performance.\footnote{Specifically: 98.0\% on Chinese HANS, 86.0\% on the stress tests. For comparison, different subsets of the English stress tests receives 85\% to 98\% agreement~\cite{naik2018stress}.  }

\subsection{Probing/diagnostic datasets}
\label{sec:prob}
While the Chinese HANS and stress tests
are designed to adversarially test
the models, we also create probing or diagnostic datasets
which are aimed at examining
the models' linguistic and reasoning abilities.

\paragraph{Hand-crafted diagnostics} \label{sec:new:diagnostics}

We expanded the diagnostic dataset from the Chinese NLU Benchmark (CLUE) \citep{clue} in the following two ways:

\begin{CJK*}{UTF8}{gbsn}
First, 6 Chinese linguists (PhD students) created diagnostics for 4 Chinese-specific linguistic phenomena. Here are two of the phenomena:\footnote{For the other two, please refer to Appendix~\ref{sec:appendix:ex}.}
(1) \textit{pro-drop}: subjects or objects in Chinese can be dropped when they can be recovered from the context \citep{li1981mandarin}. Thus the model needs to figure out the subject/object from the context.
(2) \textit{four-character idioms} (i.e., 成语~\textit{Chengyu}). They are a special type of Chinese idioms that has exactly four characters, usually with a figurative meaning different from the literary meaning, e.g., 打草惊蛇~\textit{hit hay startle snake} (behaving carelessly and causing your enemy to become vigilant). \orange{We construct examples to test whether models understand the figurative meaning in the idioms. Specifically, we first create a premise $\mathcal{P}$ which includes the idiom, where there is enough context so that a human is highly likely to interpret the idiom figuratively. Then we create an entailed hypothesis that is based on the figurative (correct) interpretation, and a neutral/contradictory hypothesis that uses the literal (incorrect) meaning (see Table~\ref{app:tab:ex:diagnostics} in the Appendix for an example). For each $\mathcal{P}$ we write 3 hypothesis, one for each inference relation.  }
We also added diagnostics involving world knowledge.
\end{CJK*}

Second, we double the number of diagnostic pairs for all 9 existing linguistic phenomena in CLUE with pairs whose
premises are selected from a large news corpus\footnote{We use the BCC corpus \cite{bcc}: \urlstyle{rm} \url{http://bcc.blcu.edu.cn/}.} and hypotheses are hand-written by our linguists, to accompany the 514 artificially created data in CLUE.
The resulting new diagnostics is 4 times as large as the original one,
with a total of 2,122 NLI pairs.
For quality control, each pair is
double-checked by local Chinese linguists not involved in this study and the controversial cases
were discarded after a discussion among the 6 linguists.
See Table~\ref{app:tab:ex:diagnostics} in Appendix~\ref{sec:appendix:ex} for examples.

\paragraph{Semantic fragments}

Following \citet{probing2020} and \citet{salvatore2019logical}, we design synthesized fragments to examine models' understanding
ability of six types of linguistic and logic
inference: \textbf{boolean}, \textbf{comparative}, \textbf{conditional}, \textbf{counting}, \textbf{negation} and \textbf{quantifier}, where
each category has 2-4 templates. See example templates and NLI pairs in Table~\ref{tab:ex:probing}.

The data is generated using context-free grammar rules
and a vocabulary of 80,000 person names (Chinese and transliterated), 8659 city names and expanded predicates and comparative relations in \citet{probing2020} to
make the data more challenging.
As a result, we generated
1,000 examples for each fragment.
For quality control, each template
was checked by 3 linguists/logicians; also 20 examples
from each category were checked for correctness by local experts.

\begin{CJK*}{UTF8}{gbsn}
\begin{table*}[t]
\scriptsize
\centering
\scalebox{.95}{
\begin{tabular}{p{1.3cm}p{8cm}p{5cm}l}\toprule
category & premise & hypothesis & label \\\midrule
Negation & 库尔图尔只到过湛江市麻章区,丰隆格只到过大连市普兰店区……
 \textit{person$_1$} only went to \textit{location$_1$}; \textit{person$_2$} only went to \textit{location$_2$}; .... & 库尔图尔没到过大连市普兰店区。\newline \textit{person$_1$} has not been to \textit{location$_2$}. & E \\
Boolean & 何峥、管得宽、李国柱……只到过临汾市襄汾县。 \newline\textit{person$_1$}, \textit{person$_2$} ... have only been to \textit{location$_1$}. & 何峥没到过遵义市红花岗区。 \newline \textit{person$_1$} has not been to \textit{location$_2$}. & E \\
Quantifier  &
有人到过每一个地方,拥抱过每一个人。\newline Someone has been to every place and hugged every person. &
王艳没拥抱过包一。  \textit{person$_1$} hasn't hugged \textit{person$_2$}. & N \\
Counting  & 韩声雄只拥抱过罗冬平、段秀芹……赵常。 \newline \textit{person$_1$} only hugged \textit{person$_2$}, \textit{person$_3$} ... \textit{person$_8$}.
 &  韩声雄拥抱过超过10个人。\newline \textit{person$_1$} hugged more than 10 people. & C \\
Conditional &
……,穆肖贝夸到过赣州市定南县, 如果穆肖贝夸没到过赣州市定南县, 那么张本伟到过呼伦贝尔市阿荣旗。... \textit{person$_n$} has been to \textit{location$_n$}. If \textit{person$_n$} hasn't been to \textit{location$_n$}, then \textit{person$_m$} has been to \textit{location$_m$}. &
张本伟没到过呼伦贝尔市阿荣旗。 \textit{person$_m$} hasn't been to \textit{location$_m$}. & N \\
Comparative  & 龙银凤比武书瑾、卢耀辉……奈德哈特都小,龙银凤和亚厄纳尔普一样大。\textit{person$_1$} is younger than  \textit{person$_2$},  ...,  \textit{person$_n$};  \textit{person$_1$} is as old as  \textit{person$_m$}  & 亚厄纳尔普比梁培娟大。  \textit{person$_m$} is older than  \textit{person$_{n-2}$}. & C \\\bottomrule
\end{tabular}}
\caption{Example NLI pairs for semantic/logic probing with translations. Each label for each category has 2 to 4 templates; we are only showing 1 template for 1 label. 1,000 examples are generated for each category. \label{tab:ex:probing}}
\end{table*}
\end{CJK*}

\section{Experimental setup}
\label{sec:exp}

Our main goal is to test
whether cross-lingual transfer are
robust against the adversarial and
probing data we created \orange{when evaluated without additional training}.
Thus we need to compare the best Chinese
monolingual models
with the best multilingual models \orange{trained either on English NLI data alone, or on combinations of Chinese and English data.}\footnote{We also run the same experiments
for Chinese-to-English transfer,
i.e., fine-tuning XLM-R with OCNLI and evaluate
on the four English counterpart datasets.
We find that transferring
from OCNLI to English does
not perform as well as monolingual
English models, likely due to the small
size of OCNLI. Detailed results are reported in Appendix~\ref{sec:app:ch2en:transfer}.}

\paragraph{Chinese monolingual models}
We experimented with two current state-of-the-art transformer models:
RoBERTa-large \citep{roberta} and Electra-large-discriminator \citep{electra}.
We use the Chinese models released from
\citep{cui2020revisiting}\footnote{We use \texttt{hfl/chinese-roberta-wwm-ext-large} from \urlstyle{rm}\url{https://github.com/ymcui/Chinese-BERT-wwm}
and \texttt{hfl/chinese-electra-large-discriminator} from  \urlstyle{rm}\url{https://github.com/ymcui/Chinese-ELECTRA}. }
implemented the Huggingface Transformer library \cite{transformers}.

\paragraph{Multilingual model} We use XLM-RoBERTa-large \citep{xlm-r}.
We choose XLM-R over mT5~\citep{mt5} because XLM-R generally performs better than mT5 under the same model size (see original paper for details). Also, XLM-R as a RoBERTa model is most related architecturally to existing Chinese pre-trained models.

\paragraph{Fine-tuning data for Chinese models \& XLM-R}
      (1) \underline{XNLI}: the full Chinese training set in the machine-translated XNLI dataset, with 390k examples \citep{xnli}.
      (2) \underline{XNLI-small}: 50k examples from XNLI, the same size as the training data of OCNLI.
      (3) \underline{OCNLI}: Original Chinese NLI dataset \citep{ocnli}. It is a Chinese NLI dataset collected
      from scratch, following the MNLI procedure, with 50k training examples. We use this to measure the effect of
      the quality of training data; that is, whether it is better to use
      small, high-quality training data (OCNLI), or large, low-quality MT data (XNLI).
      (4) \underline{OCNLI + XNLI}: a combination of the two training sets, 440k examples.

\paragraph{Fine-tuning data for XLM-R}
To examine cross-lingual transfer, we finetune XLM-R on English NLI data alone and
English + Chinese NLI data:
      (1) \underline{MNLI}: 390k examples from MNLI.train \citep{mnli}.
      (2) \underline{English all NLI}: we combine MNLI \citep{mnli}, SNLI \citep{snli}, FeverNLI \citep{fever,nie2019combining} with ANLI \citep{anli}, a total of 1,313k examples.
      (3) \underline{OCNLI + English all NLI}.
      (4) \underline{XNLI + English all NLI}. These two are set to examine whether combining Chinese and English
      fine-tuning data is helpful.

We fine-tune the models on OCNLI-dev.
Acknowledging that different training runs can produce very different checkpoints for behavioral testing \cite{d2020underspecification},  we run 5 models on different seeds and report the mean accuracy of the models with the best hyper-parameter setting (for details see Appendix~\ref{sec:app:finetune}).

\section{Results and discussion}

\subsection{Results on OCNLI\_dev}

\begin{table}[t]
\footnotesize
\centering
\scalebox{.85}{
\begin{tabular}{llcc}\toprule
Model & Fine-tuned on & Acc & Scenario \\\midrule
RoBERTa & zh MT: XNLI-small & 67.44 & monolingual \\
RoBERTa & zh MT: XNLI & 70.29 & monolingual \\
RoBERTa & zh ori: OCNLI & 79.11 & monolingual \\
RoBERTa & zh: OCNLI + XNLI & 78.43 & monolingual \\ \hdashline %
XLM-R & zh MT: XNLI & 72.55  & monolingual \\
XLM-R & zh ori: OCNLI & 79.24 & monolingual \\
XLM-R & zh: OCNLI + XNLI & \cellcolor{Gray}{80.31} & monolingual \\ \midrule %
XLM-R & en: MNLI & 71.98 & zero-shot \\
XLM-R & en: En-all-NLI & \cellcolor{Gray}{73.73} & zero-shot \\ \midrule %
XLM-R & mix: OCNLI + En-all-NLI & \cellcolor{Gray}{\textbf{82.18}} & mixed \\
XLM-R & mix: XNLI + En-all-NLI & 74.12  & mixed  \\\bottomrule
\end{tabular}
}
\caption{Results on OCNLI dev.  ``\textbf{Scenario}'' indicates whether the model is fine-tuned on Chinese \emph{only} data (\textbf{monolingual}), English data (\textbf{zero-shot}) or \textbf{mixed} English and Chinese data; results in \hlgray{gray} show best performance for each scenario. Best overall result in \textbf{bold}. Same below.  \label{tab:ocnli:dev}}
\end{table}

Results on the dev set of OCNLI are presented in Table~\ref{tab:ocnli:dev}.
For monolingual RoBERTa, we see a similar performance
as reported in the OCNLI paper \citep{ocnli},
with 79.11\% accuracy. The monolingual Electra
achieves a very close accuracy of 79.02\% (not shown
in the Table).
As we see the same trend in the following experiments,
we will therefore only report results on RoBERTa.

For XLM-R, fine-tuning on MNLI or En-all-NLI gives us
reasonable results of around 72\% to 74\%, which is better than models fine-tuned on XNLI,
indicating that fine-tuning on an English data (MNLI) alone
can outperform monolingual models fine-tuned
on the same data but machine-translated into Chinese (XNLI).\footnote{For these experiments we also tested with another Chinese machine-translated MNLI (CMNLI), translated
by a different MT system, which was released by CLUE \urlstyle{rm} (\url{https://github.com/CLUEbenchmark/CLUE}), and obtained similar results.}
This is consistent with previous results on
Korean \cite{choi2021analyzing} and
Persian \cite{parsinlu} for other NLU tasks.

What is also interesting is that
combining OCNLI and En-all-NLI gives us a boost
of 2\% to 82.18\% (a result that is comparable to the current published SOTA),
showing the power of mixing
high-quality English and Chinese training data.

\subsection{Chinese HANS}

Table \ref{hans-result} shows results of the Chinese HANS data tested on the aforementioned monolingual models and cross-lingual model.

\begin{table*}[t]
\centering
\footnotesize
\scalebox{0.85}{
\begin{tabular}{ll|c|cccc | c }
\toprule
Model & Fine-tuned on & Overall & Lexical Overlap & Sub-sequence & Entailment & Non-Entailment & $\Delta$ \\\midrule
RoBERTa                   & zh MT: XNLI-small                     & 49.48    & 58.12           & 25.42       & 99.22      & 30.26          & 37.18 \\
RoBERTa                   & zh MT: XNLI                           & 60.80    & 68.99           & 38.01       & 99.74      & 45.76          & 24.53 \\

RoBERTa                   & zh ori: OCNLI                          & \cellcolor{Gray}{71.72}    & \cellcolor{Gray}{75.39}           & \cellcolor{Gray}{61.48}       & 99.67      & \cellcolor{Gray}{60.91}          & 18.20 \\
RoBERTa                   & zh: OCNLI+XNLI                     & 69.33    & 74.73           & 54.27       & 99.89      & 57.51          & 20.92  \\
\hdashline %
XLM-R                     & zh MT: XNLI                           & 57.74    & 66.47           & 33.45       & 99.96      & 41.42          & 31.13  \\
XLM-R                     & zh ori: OCNLI                          & 61.82    & 65.83           & 50.68       & 99.89      & 47.11          & 32.13  \\
XLM-R                     & zh: OCNLI+XNLI                     & 70.31    & 74.25           & 59.34       & \cellcolor{Gray}{100.00}     & 58.84 & 21.47 \\
\midrule %
XLM-R                     & en: En-all-NLI                     & \cellcolor{Gray}{69.56}    & \cellcolor{Gray}{77.62}           & 47.13       & \cellcolor{Gray}{100.00}     & \cellcolor{Gray}{57.80}          & 15.93 \\
XLM-R                     & en: MNLI                           & 66.74    & 73.12           & \cellcolor{Gray}{48.97}       & 100.00     & 53.89          & 18.09  \\ \midrule %
XLM-R                     & mix: OCNLI+En-all-NLI              & \cellcolor{Gray}{\textbf{78.82}}    & \cellcolor{Gray}{\textbf{81.57}}           & \cellcolor{Gray}{\textbf{71.15}}       & \cellcolor{Gray}{\textbf{100.00}}     & \cellcolor{Gray}{\textbf{70.63}}          & 11.55  \\
XLM-R                     & mix: XNLI+En-all-NLI              & 66.90    & 76.25           & 40.90       & 99.93     & 41.89          & 32.23   \\\midrule
Human & & 98.00& & & &  \\
\bottomrule
\end{tabular}}
\caption{Accuracy on Chinese HANS. $\Delta =$ the difference of accuracy between OCNLI dev and Non-Entailment. %
}
\label{hans-result}
\end{table*}

\textbf{Cross-lingual transfer achieves strong results}.
We first notice that when XLM-R is fine-tuned solely on the English
data (En-all-NLI), the performance ($\sim$69\%) is only slightly
worse than the best monolingual model ($\sim$71\%).
This suggests that cross-lingual transfer from
English to Chinese is quite successful
for an adversarial dataset like HANS.
Second, adding OCNLI to En-all-NLI in the training data
gives a big boost of about 9\%, and achieves
the overall best result.
This is about 12\% higher than combining XNLI
and the English data, demonstrating
the advantage of the expert-annotated OCNLI over machine translated XNLI, even
though the latter is about 8 times the size
of the former.
Despite these results, however, we note that all models continue to perform below human performance, suggesting more room for improvement.

Our results also suggest that examples involving the \textit{sub-sequence} heuristics are more difficult than those targeting the \textit{lexical overlap} heuristics for the transformers models we tested (see the ``sub-sequence'' and ``lexical overlap'' columns in Table~\ref{hans-result}). This is in line with the results reported in the English HANS paper (specifically Table 15 in \citet{hans} which also shows that the sub-sequence examples are more difficult for the English BERT model).
Second, for the sub-sequence heuristics, results from monolingual model are 12\% higher than those from XLM-R under the zero-shot transfer setting (61.48\% versus 48.79\% in ``sub-sequence'' column in Table~\ref{hans-result}).
This stands in contrast with the lexical overlap heuristic, where the best monolingual model performs similarly to the best zero-shot cross-lingual transfer (75.39\% versus 77.62\%).
This is one of the few cases where cross-lingual transfer under-performs the monolingual setting by a large margin, suggesting that in certain situations monolingual models may be preferred.

\subsection{Stress tests}

Table~\ref{tab:stress} presents the accuracies
on all the stress tests. We first see that cross-lingual zero-shot transfer using
all English NLI data
performs even better than the best monolingual model ($\sim$74\% vs.~$\sim$71\%).
This demonstrates the power of the cross-lingual transfer-learning.
Adding OCNLI to all English NLI gives another
increase of about 3 percentage points (to 77\%), while
adding XNLI hurts the performance, again showing
the importance of having expert-annotated
language-specific data.

\begin{table*}[t]
\small
\centering
\scalebox{.82}{
\begin{tabular}{ll|c|cccccccc}
\toprule
Model   & Fine-tuned on             & Overall         & Ant. & Syn. & \parbox{1.2cm}{\centering Distr H} &
\parbox{1.2cm}{\centering Distr H-n} & \parbox{1.2cm}{\centering Distr P} &
\parbox{1.2cm}{\centering Distr P-n}  &
\parbox{1cm}{\centering Spell-\\ing} &
num.
\\\midrule
RoBERTa & zh MT: XNLI-small       & 59.41 & 43.38   & 99.64   & 51.61 & 51.41 & 70.66  & 71.19            & 69.93    & 28.70      \\
RoBERTa & zh MT: XNLI             & 66.22 & 52.28   & 99.79   &54.83 & 53.8  & 74.55  & 74.57       & 72.22    & 53.53     \\
RoBERTa & zh ori: OCNLI           & 64.49       & \cellcolor{Gray}{71.81}   & 73.66   & 52.95 & 51.8  & 73.43  & 73.86     & 71.79    & 54.16     \\
RoBERTa & zh: OCNLI + XNLI        & 71.01 & 59.39   & 99.06   & 55.87 & 54.64 & 76.83  & 76.50      & 75.48    & \cellcolor{Gray}{70.18}     \\ \hdashline %
XLM-R   & zh MT: XNLI             & 66.87 & 55.53   & \cellcolor{Gray}{\textbf{99.96}}   & 56.11 & 55.29 & 77.69  & 77.9   & 74.37    & 46.81     \\
XLM-R   & zh ori: OCNLI           & 69.08 & 71.29   & 88.63   & 55.93 & 55.05 & 76.84  & 77.00   & 71.42    & 65.51     \\
XLM-R   & zh: OCNLI + XNLI        & \cellcolor{Gray}{71.49} & 61.85   & 99.45   & \cellcolor{Gray}{58.15} & \cellcolor{Gray}{57.92} & \cellcolor{Gray}{79.16}  & \cellcolor{Gray}{79.28}     & \cellcolor{Gray}{\textbf{77.93}}    & 61.88     \\ \midrule %
XLM-R   & en:MNLI                 & 67.94 & 65.77   & \cellcolor{Gray}{99.2}    & \cellcolor{Gray}{55.14} & \cellcolor{Gray}{54.6}  & \cellcolor{Gray}{75.75}  & \cellcolor{Gray}{75.76}    & 70.76    & 50.90      \\
XLM-R   & en: En-all-NLI          & \cellcolor{Gray}{74.52}       & \cellcolor{Gray}{80.36}   & 97.58   &54.74 & 53.56 & 73.96  & 73.92    & \cellcolor{Gray}{71.02}    & \cellcolor{Gray}{82.73}     \\ \midrule %
XLM-R   & mix: OCNLI + En-all-NLI & \cellcolor{Gray}{\textbf{77.36}} & \cellcolor{Gray}{\textbf{81.93}}   & 95.09   & \cellcolor{Gray}{\textbf{59.23}} & \cellcolor{Gray}{\textbf{58.00}}    & \cellcolor{Gray}{\textbf{79.88}}  & \cellcolor{Gray}{\textbf{79.92}}  & 74.53    & \cellcolor{Gray}{\textbf{87.77}}     \\
XLM-R   & mix: XNLI + En-all-NLI  & 73.57 & 66.15   & \cellcolor{Gray}{99.68}   & 57.02 & 55.51 & 78.38  & 78.53    & \cellcolor{Gray}{75.15}    & 80.33\\\midrule
Human & &85.00 &  85.00 & 98.00 & 83.00 & 83.00 & 83.00 &  83.00 & 78.00 & 98.00 \\
\bottomrule
\end{tabular}
}
\caption{Accuracy on the stress test. Distr H/P(-n): distraction in Hypothesis/Premise (with negation).
    \label{tab:stress}}
\end{table*}

\begin{table*}[t]
\centering
\footnotesize
\scalebox{.75}{
\begin{tabular}{ll|c|ccccc|ccccccccc}
\toprule
Model &  Fine-tuned on & Overall
& \rotatebox{90}{\parbox{1.5cm}{$\ast$ Classifier}}
& \rotatebox{90}{\parbox{1.5cm}{$\ast$ Idioms}}
& \rotatebox{90}{\parbox{1.5cm}{$\ast$ Non-core\\arguement}}
& \rotatebox{90}{\parbox{1.5cm}{$\ast$ Pro-drop}}
& \rotatebox{90}{\parbox{1.5cm}{$\ast$ Time \\of event}}
& \rotatebox{90}{\parbox{1.5cm}{Anaphora}}
& \rotatebox{90}{\parbox{1.5cm}{Argument \\structure}}
& \rotatebox{90}{\parbox{1.5cm}{Common \\sense}}
& \rotatebox{90}{\parbox{1.5cm}{Comparatives}}
& \rotatebox{90}{\parbox{1.5cm}{Double \\negation}}
& \rotatebox{90}{\parbox{1.5cm}{Lexical \\semantics}}
& \rotatebox{90}{\parbox{1.5cm}{Monotonicity}}
& \rotatebox{90}{\parbox{1.5cm}{Negation}}
& \rotatebox{90}{\parbox{1.5cm}{World \\knowledge}} \\\midrule
RoBERTa & zh MT: XNLI-small & 62.9 & 65.8 & 64.7 & 55.2 & 80.5 & 60.0 & 59.6 & 67.4 & 54.3 & 61.4 & 48.3 & 60.9 & 59.7 & 66.2 & 39.0 \\
RoBERTa & zh MT: XNLI & 67.7 & {67.6} & 66.2 & {59.4} & 82.3 & 65.1 & 69.9 & 72.0 & \cellcolor{Gray}{56.8} & 70.4 & 64.2 & 67.5 & 61.7 & 72.9 & 52.1 \\
RoBERTa & zh ori: OCNLI & 67.8 & 62.0 & {68.0} & \cellcolor{Gray}{59.4} & 80.7 & {77.5} & {70.3} & 70.0 & 56.0 & 66.6 & 64.2 & 68.4 & 61.7 & 72.4 & 57.9 \\
RoBERTa & zh: OCNLI + XNLI & \ul{69.3} & 66.3 & 67.1 & 58.6 & {83.0} & 74.0 & 70.1 & {73.5} & 54.9 & {74.1} & {67.5} & {69.1} & {62.5} & {76.0} & {60.0} \\ \hdashline %
XLM-R & zh MT: XNLI & 60.9 & 61.2 & 62.3 & 50.4 & 71.9 & 59.7 & 60.3 & 63.3 & 51.7 & 65.2 & 54.9 & 61.0 & 53.5 & 66.9 & 58.3 \\
XLM-R & zh ori: OCNLI & 68.0 & 57.6 & 70.1 & 58.0 & 79.6 & 76.3 & 67.4 & 70.3 & 55.3 & 69.8 & 75.8 & 71.1 & 62.5 & 71.1 & 62.1 \\
XLM-R & zh: OCNLI + XNLI & \cellcolor{Gray}{71.5} & \cellcolor{Gray}{70.4} & \cellcolor{Gray}{71.6} & 57.5 & \cellcolor{Gray}{84.6} & \cellcolor{Gray}{77.8} & \cellcolor{Gray}{74.5} & \cellcolor{Gray}{74.7} & 55.3 & \cellcolor{Gray}{75.5} & \cellcolor{Gray}{76.7} & \cellcolor{Gray}{72.8} & \cellcolor{Gray}{62.7} & \cellcolor{Gray}{76.3} & \cellcolor{Gray}{65.3} \\ \midrule %
XLM-R & en: MNLI & 70.2 & 70.1 & 73.9 & \cellcolor{Gray}{57.5} & 86.4 & 70.8 & 69.3 & 72.9 & 48.9 & 76.0 & 62.5 & 67.8 & 62.6 & 77.0 & 62.1 \\
XLM-R & en: En-all-NLI & \cellcolor{Gray}{71.9} & \cellcolor{Gray}{71.8} & \cellcolor{Gray}{\textbf{74.3}} & 56.2 & \cellcolor{Gray}{87.4} & \cellcolor{Gray}{75.7} & \cellcolor{Gray}{74.9} & \cellcolor{Gray}{74.8} & \cellcolor{Gray}{49.1} & \cellcolor{Gray}{80.5} & \cellcolor{Gray}{70.8} & \cellcolor{Gray}{69.1} & \cellcolor{Gray}{63.8} & \cellcolor{Gray}{77.8} & \cellcolor{Gray}{64.2} \\ \midrule %
XLM-R & mix: OCNLI + En-all-NLI & \cellcolor{Gray}{\textbf{74.9}} & \cellcolor{Gray}{\textbf{72.7}} & \cellcolor{Gray}{\textbf{74.3}} & 60.1 & \cellcolor{Gray}{\textbf{88.5}} & \cellcolor{Gray}{\textbf{84.5}} & \cellcolor{Gray}{\textbf{77.3}} & \cellcolor{Gray}{\textbf{78.1}} & 56.6 & \cellcolor{Gray}{\textbf{81.3}} & \cellcolor{Gray}{\textbf{79.2}} & \cellcolor{Gray}{\textbf{77.2}} & 65.6 & \cellcolor{Gray}{\textbf{78.0}} & 67.9 \\
XLM-R & mix: XNLI + En-all-NLI & 71.4	&70.2	&58.5	& \cellcolor{Gray}{\textbf{85.5}}	&71.3	&75.2	&75.5	&55.1	& \cellcolor{Gray}{\textbf{79.2}}	&70.0	&69.1	&62.4	& \cellcolor{Gray}{\textbf{76.2}}	&72.1	& \cellcolor{Gray}{\textbf{71.3}} \\\bottomrule
\end{tabular}
}
\caption{Accuracy on the expanded diagnostics. Uniquely Chinese linguistic features
    are prefixed with $\ast$.
    \label{tab:diagnostics}}
\end{table*}

\begin{table*}[t]
\footnotesize
\centering
\scalebox{.85}{
\begin{tabular}{ll|c|cccccc}\toprule
	model & finetune on & overall & boolean & comparative & conditional & counting & negation & quantifier \\\midrule
	RoBERTa & zh MT: XNLI-small & 46.57 & 32.81 & 34.41 & 61.48 & 81.82 & 33.27 & 35.63 \\
	RoBERTa & zh MT: XNLI & 50.64 & 33.35 & 39.02 & \mygray{66.55} & 84.51 & 40.92 & 39.50 \\
	RoBERTa & zh ori: OCNLI & 47.53 & 35.81 & 34.81 & 62.87 & 69.64 & 49.84 & 32.24 \\
	RoBERTa & zh: OCNLI + XNLI & 51.13 & 38.16 & 37.98 & 66.19 & 75.73 & 53.31 & 35.43 \\ \hdashline %
	XLM-R & zh ori: OCNLI & \cellcolor{Gray}{54.33} & \cellcolor{Gray}{\textbf{54.19}} & \cellcolor{Gray}{\textbf{49.02}} & 52.46 & 79.70 & \cellcolor{Gray}{59.52} & 31.08 \\
	XLM-R & zh MT: XNLI & 50.79 & 33.39 & 35.33 & 66.01 & 87.23 & 33.17 & \cellcolor{Gray}{\textbf{49.60}} \\
	XLM-R & zh: OCNLI + XNLI & 52.43 & 34.51 & 36.93 & 59.98 & \cellcolor{Gray}{88.70} & 54.37 & 40.08 \\ \midrule %
	XLM-R & en: MNLI & 49.09 & 33.27 & 37.98 & 66.25 & 89.70 & 34.69 & 32.65 \\
	XLM-R & en: En-all-NLI & \cellcolor{Gray}{55.37} & \cellcolor{Gray}{33.43} & \cellcolor{Gray}{39.70} & \cellcolor{Gray}{66.65} & \cellcolor{Gray}{92.34} & \cellcolor{Gray}{64.11} & \cellcolor{Gray}{35.99} \\ \midrule %
	XLM-R & mix: OCNLI + En-all-NLI & \cellcolor{Gray}{\textbf{57.95}} & \cellcolor{Gray}{40.70} & \cellcolor{Gray}{44.49} & 63.67 & 91.54 & \cellcolor{Gray}{\textbf{74.47}} & 32.81 \\
	XLM-R & mix: XNLI + En-all-NLI & 57.73 & 40.30 & 37.82 & \cellcolor{Gray}{\textbf{66.67}} & \cellcolor{Gray}{\textbf{93.19}} & 61.52 & \cellcolor{Gray}{46.87} \\\bottomrule
\end{tabular}
}
\caption{Accuracy on the Chinese semantic probing datasets, designed following
\citet{probing2020}.
    \label{tab:probing}}
\end{table*}

\paragraph{Antonyms and Synonyms}

All models except those fine-tuned on OCNLI
achieved almost perfect score on the synonym
test.
However, for antonyms, both monolingual and multilingual
models fine-tuned with OCNLI perform better
than XNLI.
XLM-R fine-tuned with English NLI data only
again outperforms the best of monolingual models ($\sim$80\% vs.~$\sim$72\%).
Interestingly, adding XNLI to all English NLI
data hurts the accuracy badly (a 14\% drop),
while adding OCNLI to the same English data
improves the result slightly.

As antonyms are harder to learn \citep{breaknli}, we take our results
to mean that either expert-annotated data for Chinese or a huge English NLI dataset
is needed for a model to learn decent
representations about antonyms, as indicated by the high performance of RoBERTa fine-tuned with OCNLI (71.81\%), and XLM-R fine-tuned with En-all-NLI (80.36\%), on antonyms.  That is,
using machine-translated XNLI will not
work well for learning antonyms ($\sim$55\% accuracy).

\paragraph{Distraction}

Results in Table~\ref{tab:stress} show that adding distractions to the hypotheses has a more negative
negative impact on models' performance, compared with
appending distractions to the premises.
The difference is about 20\% for all models (see  ``Distr H'' columns and ``Distr P'' columns in Table~\ref{tab:stress}),
which has not been reported in previous studies,
to the best of our knowledge.
Including a negation in the hypothesis makes it
even more challenging, as we see another
one percent drop in the accuracy for all models.
This is expected as previous literature
has demonstrated the key role negation
plays when hypotheses are produced by the annotators~\citep{poliak2018hypothesis}.

\paragraph{Spelling}
This is another case where cross-lingual transfer
with English data alone falls behind
monolingual Chinese models (by about 4\%).
Also
the best results are from fine-tuning XLM-R with
OCNLI + XNLI, rather than a combination
of English and Chinese data.
Considering the data is created by swapping
Chinese characters with others
of the same pronunciation, we take it
to suggest that monolingual models
are still better at picking up
the misspellings or learning the connections
between characters at
the phonological level.

\paragraph{Numerical Reasoning}
Results in the last column of Table~\ref{tab:stress} suggest a similar pattern:
using all English NLI data for cross-lingual
transfer outperforms the best monolingual model.
However, fine-tuning
a monolingual model with the small OCNLI (50k examples, accuracy: 54\%) achieves better accuracy
than using a much larger MNLI (390k examples, accuracy: 51\%) for cross-lingual
transfer, although both are worse than XLM-R fine-tuned with all English NLI which has more than 1,000k examples (accuracy: 83\%).
This suggests that there are cases where
a monolingual setting (RoBERTa with OCNLI) is competitive against
zero-shot transfer with a large
English dataset (XLM-R with MNLI). However, that competitiveness
may disappear when the English dataset grows
to an order of magnitude larger in size or becomes more diverse (En-all-NLI contains 4 different English NLI datasets).

\subsection{Hand-written diagnostics}

Results on the expanded diagnostics are presented in Table~\ref{tab:diagnostics}.
We first see that XLM-R fine-tuned with only English
performs very well, at 70.2\% and 71.9\%, even slightly
higher than the best monolingual Chinese model (69.3\%).

Most surprisingly, \textbf{in 3/5 categories with uniquely
Chinese linguistic features,
zero-shot transfer outperforms
monolingual models}.
Only in ``non-core arguments'' and ``time of event'' do we see
higher performance of OCNLI as the fine-tuning data.
What is particularly striking is that for ``idioms (\textit{Chengyu})'', XLM-R fine-tuned only on English data achieves the best result, suggesting that
the cross-lingual transfer is capable of learning meaning representation beyond the surface lexical information, at least for many of the idioms we tested. The overall results (accuracy of 74.3\%) indicate that cross-lingual transfer is very successful in most cases.
Manual inspection of the results shows that for many NLI pairs with idioms, XLM-R correctly predicts the figurative interpretation of the idiom as entailment, and the literal interpretation as non-entailment, as described in section~\ref{sec:new:diagnostics}.
Looking at OCNLI and XNLI, we observe that
they perform similarly when fine-tuned on monolingual RoBERTa. \orange{However, when fine-tuned with XLM-R, OCNLI has a clear advantage (68.0\% versus 60.9\%), suggesting that OCNLI may produce more stable results than XNLI.} Furthermore, when coupled
with English data to be used with XLM-R, we see
again that OCNLI + En-all-NLI results in an accuracy 3 percent higher than XNLI + En-all-NLI.

\subsection{Semantic fragments}

Results on the semantic probing datasets (shown in Table~\ref{tab:probing}) are more mixed.
First, the results are in general much
worse than the other evaluation data,
but overall, XLM-R fine-tuned with OCNLI and
all English data still performs the best.
The overall lower performance is likely due to the longer length of premises and hypotheses in the semantic probing datasets, compared with the other three evaluation sets.
Second, zero-shot transfer is better
or on par with monolingual Chinese RoBERTa in 4/6 semantic fragments (except Boolean and quantifier).
Third, for Boolean and comparative,
XLM-R fine-tuned with OCNLI has a much
better result than all other monolingual models or XLM-R fine-tuned with mixed data.
We also observe that all models have highest performance on the ``counting'' fragment. Note that none of the models have seen any data from the ``counting'' fragment during fine-tuning. That is, all the knowledge come from the pre-training and fine-tuning on general NLI datasets.
The surprisingly good performance of XLM-R model (w/ En-all-NLI, 92.34\%) suggests that it may have already acquired a mapping from counting the words/names to numbers, and this knowledge can be transferred cross-linguistically.

\section{Conclusion and Future Work}

In this paper, we examine the cross-lingual transfer ability of XLM-R in the context of
Chinese NLI through four new sets of aversarial/probing tasks consisting of a total of 17 new high-quality and linguistically motivated challenge datasets. We find that cross-lingual transfer via fine-tuning solely on benchmark English data generally yields impressive performance. In 3/4 of our task categories, such \emph{zero-shot transfer} outperforms our best monolingual models trained on benchmark Chinese NLI data, including 3/5 of our hand-crafted challenge tasks that test uniquely Chinese linguistic phenomena. These results suggest that multilingual models are indeed capable of considerable cross-lingual linguistic transfer and that zero-shot NLI may serve as a serious alternative to large-scale dataset development for new languages.

These results come with several important caveats. Model performance is still outperformed by conservative estimates of human performance and our best models still have considerable room for improvement; we hope that our new resources will be useful for continuing to benchmark progress on Chinese NLI. We also find that high-quality Chinese NLI data (e.g., OCNLI) \emph{can} help improve results further, which suggests an important role for certain types of expertly annotated monolingual data in a training pipeline. In virtue of our study being limited to \emph{behavioral testing}, the exact reason for \emph{why} cross-lingual zero-shot transfer generally performs well, especially on some Chinese-specific phenomena, is an open question that requires further investigation. In particular, we believe that techniques that couple behavioral testing with \emph{intervention} techniques \cite{geiger-etal-2020-neural,vig2020causal} and other analysis methods \cite{giulianelli2018under,belinkov2019analysis,sinha2021masked} might provide insight, and that our new resources can play an important role in such future work.

\section*{Acknowledgments}
This research was supported in part by Lilly Endowment, Inc., through its support for the Indiana University Pervasive Technology Institute.
He Zhou is sponsored by China Scholarship Council.