\section{Introduction}
Named Entity Recognition (NER) \cite{Sundheim1995NamedET} has attracted increasing attention in natural language processing community. It has been applied to a lot of domains such as news \citep{tjong-kim-sang-2002-introduction,tjong-kim-sang-de-meulder-2003-introduction}, E-commerce \citep{10.1145/3404835.3463102}, social media \cite{strauss-etal-2016-results,derczynski-etal-2017-results} and bio-medicine \citep{dougan2014ncbi,li2016biocreative}. Several recent studies focus on improving the accuracy of NER models through utilizing image information (MNER) in tweets \citep{zhang2018adaptive,moon-etal-2018-multimodal,lu-etal-2018-visual}. Most approaches to MNER use the attention mechanism to model the interaction between image and text representations \citep{yu-etal-2020-improving-multimodal,zhang2021multi,Sun2021RpBERTAT}, in which image representations are from a pretrained feature extractor, i.e. ResNet \citep{he2016deep}, and text representations are extracted from pretrained textual embeddings, i.e. BERT \citep{devlin-etal-2019-bert}. Since these models are separately trained on datasets of different modalities and their feature representations are not aligned, it is difficult for the attention mechanism to model the interaction between the two modalities.

Recently, pretrained vision-language (V+L) models such as LXMERT \citep{Tan2019LXMERTLC}, UNITER \citep{chen2020uniter} and Oscar \citep{li2020oscar} have achieved significant improvement on several cross-modal tasks such as image captioning, VQA \citep{Agrawal2015VQAVQ}, NLVR \citep{Young2014FromID} and image-text retrieval \citep{Suhr2019ACF}. Most pretrained V+L models are trained on image-text pairs and simply concatenate text features and image features as the input of pretraining. There are, however, two problems. First, texts in these datasets mainly contain common nouns instead of named entities\footnote{\url{https://visualgenome.org/data_analysis/statistics}} which leads to an inductive bias over common nouns and images. Second, despite its important role in pretraining V+L models,  the image modality only plays an auxiliary role in MNER for disambiguation, and can sometimes even be discarded. These problems make pretrained V+L models perform weaker than pretrained language models for MNER.

Pretrained textual embeddings such as BERT, XLM-RoBERTa \citep{conneau-etal-2020-unsupervised} and LUKE \citep{yamada-etal-2020-luke} have achieved state-of-the-art performance on various NER datasets through simple fine-tuning of pretrained textual embeddings. Since most of the transformer-based pretrained textual embeddings are trained over long texts, recent work \citep{akbik-etal-2019-pooled,schweter2020flert,yamada-etal-2020-luke,wang-etal-2021-improving} has shown that introducing document-level contexts can significantly improve the accuracy of a NER model. The attention mechanism in transformer-based pretrained textual embeddings can utilize contexts to improve the token representation of a sequence. Moreover, pretrained V+L models such as Oscar and VinVL \citep{zhang2021vinvl} can use object tags detected in images to significantly ease the alignments between text and image features. Therefore, the images in MNER can be converted to texts as well so that the image representations can be aligned to the space of text representations. As a result, the attention module of the pretrained textual embeddings have the capability to easily model the interactions between aligned image and text representations, without introducing a new attention module. In this paper, we propose ITA, a simple but effective framework for {\bf I}mage-{\bf T}ext {\bf A}lignments. ITA converts an image into visual contexts in textual space by multi-level alignments. We concatenate the NER texts with the visual contexts as a new cross-modal input view and then feed it into a pretrained textual embedding model to improve the token representations of NER texts, which are fed into a linear-chain CRF \citep{10.5555/645530.655813} layer for prediction.
In practice, a MNER model should be robust when there is only text information, as images may be unavailable or can introduce noises. Sometimes it is even undesirable to use images as image feature extraction can be inefficient in online serving. Therefore, we further propose to utilize the cross-modal input view to improve the accuracy of textual input view, based on cross-view alignment that minimizes the KL divergence over the probability distributions of the two views.

ITA can be summarized in four aspects:
\begin{enumerate}[leftmargin=*]
    \item Object Tags as Local Alignment: ITA locally extracts object tags and its corresponding attributes of image regions from an object detector.
    \item Image Captions as Global Alignment: ITA summarizes what the image is describing through predicting image captions from an image captionilng model.
    \item Optical Character Alignment: ITA extracts the texts presented in the image via optical character recognition (OCR).
    \item Cross-View Alignment: we calculate the KL divergence between the output distributions of two input views.
\end{enumerate}
We show in experiments that ITA can significantly improve the model accuracy on MNER datasets and achieve the state-of-the-art. The cross-view alignment module can significantly improve both the cross-modal and textual input views, and bridge the performance gap between the two views.

\begin{figure*}[ht]
	\centering
	\includegraphics[scale=0.61]{model.pdf}
	\caption{The architecture of ITA. ITA aligns an image into object tags, image captions and texts from OCR. ITA takes them as visual contexts and then feeds them together with the input texts into the transformer-based embeddings. In the cross-view alignment module, ITA minimizes the distance between the output distribution of cross-modal inputs and textual inputs.
	}
	\label{fig:architecture}
\end{figure*}

\section{Approaches}
We consider the NER task as a sequence labeling problem. Given a sentence ${\bm{w}} = \{w_1, \cdots, w_n\}$ with $n$ tokens and its corresponding image $I$, an sequence labeling model aims to predict a label sequence ${\bm{y}} = \{y_1, \cdots, y_n\}$ at each position.
In our framework, we focus on incorporating visual information to improve the representations of the input tokens by aligning visual and textual information effectively. We use a visual context generator to convert the image $I$ into texts forming visual contexts ${\bm{w}}^\prime = \{w_1^\prime, \cdots, w_m^\prime\}$ with $m$ tokens. We then concatenate the input text and visual contexts as a cross-modal text+image (\textbf{I+T}) input view instead of the text (\textbf{T}) input view. We feed the \textbf{I+T} input into a pretrained textual embeddings model to get stronger token representations of the input sentence. Then the token representations are fed into a linear-chain CRF layer to get the label sequence ${\bm{y}}$. To further improve the model accuracy of both input views, we use the cross-view alignment module to align the output distributions of \textbf{I+T} and \textbf{T} input views during training. The architecture of our framework is shown in Figure \ref{fig:architecture}.

\subsection{NER Model Architecture}
We use a neural model with a linear-chain CRF layer, a widely used approach for the sequence labeling problem \citep{Huang2015BidirectionalLM,akbik-etal-2018-contextual,devlin-etal-2019-bert}. The input is fed into a transformer-based pretrained textual embeddings model and the output token representations $\{{\bm{r}}_1, \cdots, {\bm{r}}_n\}$ are fed into the CRF layer:
\begin{align}
    p_\theta({\bm{y}}|{\bm{w}}) &= \frac{\prod\limits_{i=1}^{n} \psi(y_{i-1}, y_i, {\bm{r}}_i)}{\sum\limits_{{\bm{y}}' \in \mathcal{Y}({\bm{w}})} \prod\limits_{i=1}^{n} \psi(y'_{i-1}, y'_i, {\bm{r}}_i)}\nonumber
\end{align}
where $\theta$ is the model parameters, $\mathcal{Y}({\bm{w}})$ is the set of all possible label sequences given the input ${\bm{w}}$. Given the gold label sequence $\hat{{\bm{y}}}$ in the training data, the objective function of the model for the \textbf{T} input view is:
\begin{align}
\mathcal{L}_\text{T}(\theta) = - \log p_\theta(\hat{{\bm{y}}}|{\bm{w}}) \label{eq:nll_loss}
\end{align}
The loss can be calculated using Forward algorithm.

\subsection{Image-text Alignments}
The transformer-based pretrained textual embeddings have strong representations over texts. Therefore, ITA converts the image information into textual space through generating texts from the image so that the learning of the self-attention in the transformer-based model can be significantly eased compared with simply using image features from an object detector. We propose a local (LA), a global (GA) and an optical character alignment (OCA) approaches for alignments.
\paragraph{Object Tags as Local Alignment} Given an image, the image information can be decomposed into a set of objects in local regions. The object tags of each region textually describe the local information in the image. To extract the objects, we use an object detector $\textbf{OD}$ to identify and locate the objects in the image:
\begin{align}
&{\bm{a}}, {\bm{o}} = \textbf{OD}(I); \text{where }  \nonumber\\
&{\bm{a}} = \{{\bm{a}}_1,{\bm{a}}_2,\cdots,{\bm{a}}_l\} \text{ and } {\bm{o}} = \{{o}_1,{o}_2,\cdots,{o}_l\} \nonumber
\end{align}
The attribute predictions from the object detector contain multiple attribute tags ${\bm{a}}_i$ for each object ${o}_i$. We linearize and sort the objects in a descending order based on the confidences of the detection model. For each object, we heuristically keep 0 to 3 attributes with confidence scores above a threshold $m$. We linearize the attributes and put the attributes before the corresponding objects since the attributes are the adjectives describing the object tags.
As a result, we take the predicted $l$ object tags ${\bm{o}}$ and their attribute tags ${\bm{a}}$ from the object detector as the locally aligned visual contexts ${\bm{w}}^{\text{LA}}$:
\begin{align}
{\bm{w}}^{\text{LA}} = \{{\bm{a}}_1,{o}_1,{\bm{a}}_2,{o}_2,\cdots,{\bm{a}}_l,{o}_l\}\nonumber
\end{align}

\paragraph{Image Captions as Global Alignment} Though the local alignment can localize the image into objects, the objects cannot fully describe the of the whole image. Image captioning is a task that predicts the meaning of an image. Therefore, we align the image into $k$ image captions by an image captioning model $\textbf{IC}$:
\begin{align}
\{{\bm{w}}^1,{\bm{w}}^2,\cdots,{\bm{w}}^k\} = \textbf{IC}(I) \nonumber
\end{align}
where $\{{\bm{w}}^1,{\bm{w}}^2,\cdots,{\bm{w}}^k\}$ are captions generated from beam search with $k$ beams. We concatenate the $k$ captions together with a special separate token [X] to form the aligned global visual contexts ${\bm{w}}^{\text{GA}}$:
\begin{align}
 {\bm{w}}^{\text{GA}}  {=} [{\bm{w}}^1,\text{[X]},{\bm{w}}^2,\text{[X]},\cdots,\text{[X]},{\bm{w}}^k] \nonumber
\end{align}
The exact label (e.g. ``[SEP]'' in BERT) of the special [X] token depends on the selection of embeddings.

\paragraph{Optical Character Alignment} Some image contain text when they are created to enrich the semantic information that the images want to convey. In order to better understand this type of image, we use an \textbf{OCR} model to identify and extract the texts in the image:
\begin{align}
{\bm{w}}^{\text{OCA}} = \textbf{OCR}(I) \nonumber
\end{align}
where ${\bm{w}}^{\text{OCA}}$ are the texts extracted by the \textbf{OCR} model. Note that ${\bm{w}}^{\text{OCA}}$ may be an empty text if there is no text in the image.

We concatenate the input sentence and our aligned visual contexts to form the \textbf{I+T} input view $\hat{{\bm{w}}}=[{\bm{w}};{\bm{w}}^\prime]$, where ${\bm{w}}^\prime$ can be one of ${\bm{w}}^{\text{LA}}$, ${\bm{w}}^{\text{GA}}$, ${\bm{w}}^{\text{OCA}}$ or the concatenation of all (we denote it as \textbf{All}). The transformer-based embeddings are fed with the \textbf{I+T} input view and then output image-text fused token representations for each token $\{{\bm{r}}_1^\prime, \cdots, {\bm{r}}_n^\prime\}$. The token representations are fed into the CRF layer to get the probability distribution $p_\theta({\bm{y}}|\hat{{\bm{w}}})$. Similar to Eq. \ref{eq:nll_loss}, the objective function of the model for the \textbf{I+T} input view is:
\begin{align}
\mathcal{L}_{\text{I+T}}(\theta) = - \log p_\theta(\hat{{\bm{y}}}|\hat{{\bm{w}}}) \label{eq:nll_x}
\end{align}
\paragraph{Cross-View Alignment}
There are several limitations in incorporating images into NER prediction: 1) the images may not available in testing; 2) aligning images to texts requires several pipelines in pre-processing instead of an end-to-end manner, which is so time-consuming that it is not applicable to some time-critical scenes such as online serving; 3) the noises in the image can mislead the MNER model to make wrong predictions. To alleviate these issues, we propose Cross-View Alignment (CVA), which targets at reducing the gap between the \textbf{I+T} and \textbf{T} input views over the output distributions so that the MNER model can better utilize the textual information in the input.
During training, CVA minimizes the KL divergence over the probability distribution of \textbf{I+T} and \textbf{T} input views:
\begin{align}
\mathcal{L}_{\text{CVA}}(\theta)&{=}\text{KL}(p_\theta({\bm{y}}|\hat{{\bm{w}}})||p_\theta({\bm{y}}|{\bm{w}})) \label{eq:mda_loss}
\end{align}
Since the \textbf{I+T} input view has additional visual information in the input and we want the \textbf{T} input view to match the accuracy of \textbf{I+T} input view, we only back-propagate through $p_\theta({\bm{y}}|{\bm{w}})$ in Eq. \ref{eq:mda_loss}. Therefore, Eq. \ref{eq:mda_loss} is equivalent to calculating the cross-entropy loss over the two distributions:
\begin{align}
\mathcal{L}_{\text{CVA}}(\theta)&{=}\sum_{{\bm{y}} \in \mathcal{Y}({\bm{x}})} p_\theta({\bm{y}}|\hat{{\bm{w}}}) \log p_\theta({\bm{y}}|{\bm{w}}) \label{eq:xe_loss}
\end{align}
As the set of all possible label sequences $ \mathcal{Y}({\bm{x}})$ is exponential in size, we calculate the posterior distributions of each position $p_\theta({y}_i|{\bm{w}})$ and $p_\theta({y}_i|\hat{{\bm{w}}})$ through forward-backward algorithm to approximate Eq. \ref{eq:xe_loss}:
\begin{align}
&p_\theta({y}_k|*) {\propto} \sum\limits_{\{y_0,\dots,y_{k-1}\}} {\prod\limits_{i=1}^{k}} \psi(y_{i-1}, y_i, {\bm{r}}^*_i) \nonumber\\
&{\times} \sum\limits_{\{y_{k{+}1},\dots,y_n\}} {\prod\limits_{i{=}k{+}1}^{n}} \psi(y_{i{-}1}, y_i, {\bm{r}}^*_i) \nonumber \\
&\mathcal{L}_{\text{CVA}}(\theta){=}\sum_{i=1}^n p_\theta({y}_i|\hat{{\bm{w}}}) \log p_\theta({y}_i|{\bm{w}})) \label{eq:mda}
\end{align}
where ${\bm{r}}^*_i$ represents either ${\bm{r}}_i$ or ${\bm{r}}_i^\prime$.
\paragraph{Training} During training, we jointly train \textbf{T} and \textbf{I+T} input views with the training objective in Eq. \ref{eq:nll_loss} and \ref{eq:nll_x} together with the CVA alignment training objective in Eq. \ref{eq:mda}. As a result, the final training objective for ITA is:
\begin{align}
\mathcal{L}_{\text{ITA}} = \mathcal{L}_{\text{CVA}} + \mathcal{L}_{\text{T}} + \mathcal{L}_{\text{I+T}} \nonumber
\end{align}

\section{Experiments}
\label{sec:exp}
We conduct experiments on three MNER datasets. To show the effectiveness of our approaches, we use two embedding settings and compare our approaches with previous multi-modal approaches.
\subsection{Settings}

\paragraph{Datasets}
We show the effectiveness of our approaches on Twitter-15, Twitter-17 and SNAP Twitter datasets\footnote{Twitter-15 and 17 datasets are available at \url{https://github.com/jefferyYu/UMT}.} containing 4,000/1,000/3,357, 3,373/723/723 and 4,290/1,432/1,459 sentences in train/development/test split respectively. The Twitter-15 dataset is constructed by \citet{zhang2018adaptive}. The SNAP dataset is constructed by \citet{lu-etal-2018-visual} and the Twitter-17 dataset is a filtered version of SNAP constructed by \citet{yu-etal-2020-improving-multimodal}.

\paragraph{Model Configuration}
For token representations, we use BERT base model to fairly compare with most of the recent work \citep{yu-etal-2020-improving-multimodal,zhang2021multi,Sun2021RpBERTAT}. Recently, XLM-RoBERTa has achieved state-of-the-art accuracy on various NER datasets by feeding the input together with contexts to the model. To further utilize the visual contexts in transformer-based embeddings, we use XLM-RoBERTa large (XLMR) model as another embedding in our experiments.
To extract object tags and image captions of the image, we use VinVL \citep{zhang2021vinvl}, which is a pretrained V+L model based on a newly pretrained large-scale object detector based on the ResNeXt-152 C4 architecture. We use the object detection module of VinVL to predict object tags and their corresponding attributes. The number of object tags and attributes varies over the images and is no more than $100$. We set the threshold $m$ to be $0.1$ for keeping the attributes of each object. For image captions, we use VinVL large model finetuned on MS-COCO \citep{lin2014microsoft} captions\footnote{\url{github.com/microsoft/Oscar}} with CIDEr optimization \citep{rennie2017self}. In our experiments, we use a beam size of $5$ with at most 20 tokens for prediction and keep all the $5$ captions as the visual contexts. For OCR, we use Tesseract OCR\footnote{\url{github.com/tesseract-ocr/tesseract}} \citep{smith2007overview}, which is an open source OCR engine. We use the default configuration of the engine to extract texts in the image\footnote{Please refer to Appendix \ref{app:oca} for more statistics.}.

\paragraph{Training Configuration}
During training, we finetune the pretrained textual embedding model by AdamW \citep{loshchilov2018decoupled} optimizer. In experiments we use the grid search to find the learning rate for the embeddings within $[1\times 10^{-6}, 5\times 10^{-4}]$. For BERT embeddings, we finetune the embeddings with a learning rate of $5\times 10^{-5}$ with a batch size of $16$. For XLMR embeddings, we use a learning rate of $5\times 10^{-6}$ and a batch size of $4$ instead. For the learning rate of the CRF layer, we use a grid search over $[0.05,0.5]$ and $[0.005,0.05]$ for BERT and XLMR respectively. The MNER models are trained for $10$ epochs and we report the average results from $5$ runs with different random seeds for each setting.

\begin{table}[t!]
\small
\centering
\setlength\tabcolsep{1pt}
\begin{tabular}{c|l||cc|cc|cc}
\hlineB{4}
\multicolumn{2}{c||}{} & \multicolumn{2}{c|}{Twitter-15} & \multicolumn{2}{c|}{Twitter-17} & \multicolumn{2}{c}{SNAP} \\
 \hline
\multirow{3}{*}{\makecell{Train\\Modal}} & \multirow{3}{*}{Approach} & \multicolumn{2}{c|}{\bf Eval} & \multicolumn{2}{c|}{\bf Eval} & \multicolumn{2}{c}{\bf Eval}\\
 & & \multicolumn{2}{c|}{\bf Modal} & \multicolumn{2}{c|}{\bf Modal} & \multicolumn{2}{c}{\bf Modal}\\
&  & {\textbf{T}} & {\textbf{I+T}} & {\textbf{T}} & {\textbf{I+T}} & {\textbf{T}} & {\textbf{I+T}}\\
\hline\hline
 \multicolumn{8}{c}{\bf \textsc{BERT-CRF}}\\
\hline
T & {\textbf{BERT-CRF}} & 74.79 & - & 85.18 & - & 85.98 & -\\
\hline
\multirow{7}{*}{I+T} & {\textbf{ITA-LA}} & - & 75.18 & - & 85.67& - & 86.26\\
 & {\textbf{ITA-GA}} & - & 75.17  & - & 85.75  & - & 86.72\\
 & {\textbf{ITA-OCA}} & - & 75.01 & - & 85.64& - & 86.52\\
 & {\textbf{ITA-All}} & - & 75.15 & - & 85.78 & - & 86.79\\
 & {\textbf{ITA-LA$_{\text{+CVA}}$}} & 75.26 & 75.20 & 85.72 & 85.62& 86.51 & 86.41\\
 & {\textbf{ITA-GA$_{\text{+CVA}}$}} & 75.45 & 75.52 & 85.96 & \textbf{85.85}& 86.42 & 86.39\\
 & {\textbf{ITA-OCA$_{\text{+CVA}}$}} & 75.26 & 75.30 & 85.73 & 85.79& 86.64 & 86.59\\
 & {\textbf{ITA-All$_{\text{+CVA}}$}} & \textbf{75.67} & \textbf{75.60} & \textbf{85.98} & 85.72&  \textbf{86.83} & \textbf{86.75} \\
\hline
\multicolumn{8}{c}{\bf \textsc{XLMR-CRF}}\\
\hline
T & {\textbf{XLMR-CRF}} & 77.37 & - & 88.73 & - & 89.39 & - \\
\hline
\multirow{7}{*}{I+T} & {\textbf{ITA-LA}} & - & 77.64 & - & 89.29& - & 89.68\\
 & {\textbf{ITA-GA}} & - & 77.78 & - & 89.32& - & 89.78 \\
& {\textbf{ITA-OCA}} & - & 77.94 & - & 89.31& - & 89.64\\
& {\textbf{ITA-All}} & - & 77.81 & - & 89.62 & - & 90.10\\
& {\textbf{ITA-LA$_{\text{+CVA}}$}} & 77.87 & 77.93 & 89.45 & \textbf{89.90}& 89.85 & 89.91\\
 & {\textbf{ITA-GA$_{\text{+CVA}}$}} & 78.03 & 78.02 & 89.41 & 89.62& 89.85 & 90.09 \\
& {\textbf{ITA-OCA$_{\text{+CVA}}$}} & 77.57 & 77.59 & 89.32 &89.55& 89.90 & 89.84\\
& {\textbf{ITA-All$_{\text{+CVA}}$}} & \textbf{78.25} & \textbf{78.03} & \textbf{89.47} & 89.75& \textbf{90.02} & \textbf{90.15}\\
\hlineB{4}
\end{tabular}
\caption{A comparison of ITA and our baseline.}
\label{tab:main}
\end{table}

\begin{table}[t!]
\small
\centering
\begin{tabular}{l||c|c|c}
\hlineB{4}
\multicolumn{1}{c||}{Approach} & \multicolumn{1}{c|}{Twitter-15} & \multicolumn{1}{c|}{Twitter-17} & \multicolumn{1}{c}{SNAP} \\
\hline
\multicolumn{4}{c}{\bf \textsc{Reported F1 of Previous Approaches}}\\
\hline
{\textbf{BERT-CRF}}$^\dagger$ & 71.81 &  83.44 & -\\
{\textbf{OCSGA}}$^\clubsuit$ & 72.92 & - & - \\
{\textbf{UMT}}$^\dagger$ & 73.41 & 85.31 & -\\
{\textbf{RIVA}}$^\ddagger$ & 73.80 & - & 86.80 \\
{\textbf{RpBERT$_{\text{base}}$}}$^\spadesuit$ &74.40 & - & 87.40 \\
{\textbf{UMGF}}$^\diamond$ & 74.85 & 85.51 & - \\
\hline
\multicolumn{4}{c}{\bf \textsc{Our Reproductions}}\\
\hline
{\textbf{BERT-CRF}} & 74.79 & 85.18 & 85.98\\
{\textbf{UMT}} & 72.83 & 84.88 & - \\
{\textbf{UMGF}} & 74.42 & 85.27 & - \\
{\textbf{RpBERT$_{\text{base}}$}} & 67.21 & - & 62.14 \\
{Ours: \textbf{ITA-All$_{\text{+CVA}}$}} & \textbf{76.01} & \textbf{86.45} & \textbf{87.44} \\
\hlineB{4}
\end{tabular}
\caption{A comparison of our approaches and state-of-the-art approaches. $\clubsuit$: \citet{10.1145/3394171.3413650}; $\dagger$: results are from \citet{yu-etal-2020-improving-multimodal}; $\ddagger$: \citet{sun-etal-2020-riva}, $\spadesuit$: \citet{Sun2021RpBERTAT}, note that {\textbf{RpBERT$_{\text{base}}$}} uses the test set to select the best model; $\diamond$: results are from \citet{zhang2021multi}.}
\label{tab:previous_sota}
\end{table}

\subsection{Results}
In Table \ref{tab:main}, we compare our approaches with our baselines with different training and evaluation modalities (\textbf{T} for the text-only input view and \textbf{I+T} for the multi-modal input view). Results show that ITA models are significantly stronger than our {\textbf{BERT-CRF}} and {\textbf{XLMR-CRF}} baselines (Student's t-test with $p<0.05$). For the aligned visual contexts, LA, GA and OCA are competitive in most of the cases. To show the effectiveness of CVA, we report the evaluation results of both input views in evaluation. With CVA, the accuracy of both input views can be improved, especially the \textbf{T} input view. CVA can improve the \textbf{T} input view to be competitive with \textbf{I+T} input view. Moreover, the combination of all the alignments \textbf{ITA-All$_{\text{+CVA}}$} can further improve the model accuracy in most of the cases. The accuracy of the MNER models can be significantly improved if we use XLMR embeddings, which shows the importance of the text modality in MNER. With XLMR embeddings, the model accuracy can be further improved with ITA. The relative improvements over the baseline models are sometimes higher with XLMR than with BERT, which shows that the visual contexts can be further utilized with stronger embeddings.

In Table \ref{tab:previous_sota}, we compare \textbf{ITA} with previous state-of-the-art approaches. For previous approaches, we report the results including
\textbf{OCSGA}, \textbf{UMT}, \textbf{RIVA}, \textbf{RpBERT}, \textbf{UMGF}, which are the proposed approaches of \citet{10.1145/3394171.3413650}, \citet{yu-etal-2020-improving-multimodal}, \citet{sun-etal-2020-riva}, \citet{Sun2021RpBERTAT} and \citet{zhang2021multi} respectively. For fair comparison, we report the results of these models based on the BERT base embeddings. Moreover, since most of these previous approaches report the best model accuracy instead of the averaged model accuracy, we use the best model accuracy of \textbf{ITA-All$_{\text{+CVA}}$} over $5$ runs. We also report our reproduced results of \textbf{UMT}, {\textbf{RpBERT}} and \textbf{UMGF} on the corresponding datasets.
The results show that \textbf{ITA-All$_{\text{+CVA}}$} outperforms all of the previous approaches. On the SNAP dataset, the reported accuracy of {\textbf{RpBERT$_{\text{base}}$}} is competitive with \textbf{ITA-All$_{\text{+CVA}}$}. However, we find that the accuracy of our reproduced {\textbf{RpBERT$_{\text{base}}$}}\footnote{We reproduced the results based on the official code for {\textbf{RpBERT$_{\text{base}}$}}: \url{https://github.com/Multimodal-NER/RpBERT}} is significantly lower than the reported accuracy, even after careful check of the source code and hyper-parameter tuning.
Moreover, the fact that our \textbf{BERT-CRF} baseline achieves competitive accuracy with previous state-of-the-art multi-modal approaches shows that most of the previous work has not fully explored the strength of the text representations for the task.

\begin{table}[t!]
\small
\centering
\setlength\tabcolsep{4pt}
\begin{tabular}{l||c|c}
\hlineB{4}
Approaches & \multicolumn{1}{c|}{Twitter-15} & \multicolumn{1}{c|}{Twitter-17} \\%& \multicolumn{1}{c}{SNAP} \\
\hline\hline
{\textbf{BERT-CRF$_{\text{UMT}}$}} & 71.81 &  83.44 \\% & - \\
{\textbf{BERT-CRF$_{\text{Ours}}$}} & 74.79 & 85.18 \\% & 85.98\\
\hline
\multicolumn{3}{c}{\bf \textsc{Our Reproductions}}\\
\hline
{\textbf{BERT-CRF$_{\text{UMT}}$}} & 71.74 & 84.20 \\%& -\\
{\textbf{BERT-CRF$_{\text{UMT-Improved}}$}} & 72.53 & 84.48 \\%& -\\
{\textbf{UMT}} & 72.83 & 84.88 \\%& -\\
{\textbf{UMT$_{\text{Improved}}$}}  & 72.96 & 84.50 \\%& -\\
\hlineB{4}
\end{tabular}
\caption{Our reproductions of previous baselines and approaches. ``Improved'' means our improved models based on the UMT code base.}
\label{tab:reproduction}
\end{table}

\paragraph{Discussion about Textual Modules}
As we have shown in Table \ref{tab:main} and \ref{tab:previous_sota}, the textual baselines (i.e. {\textbf{BERT-CRF}}) of previous work are significantly lower than that of ours. In most of the previous MNER architectures, the textual modules are mainly based on the baseline architectures with some modifications. We further show the baselines of previous work are not well-trained and how the multi-modal approaches perform with stronger textual modules. In Table \ref{tab:reproduction}, we rerun the \textbf{BERT-CRF} baseline based on the released codes of \textbf{UMT}\footnote{\url{https://github.com/jefferyYu/UMT}}. Based on the code of \textbf{UMT}, we tried to improve the baseline models in the code by using the same loss function as ours\footnote{The details are discussed in Appendix \ref{app:bug}}. The accuracy of \textbf{BERT-CRF} models in the code are significantly improved but the \textbf{UMT} models based on the improved code are not improved and even get worse in Twitter-17. Therefore, we suspect the \textbf{UMT} model cannot be further improved even with stronger textual modules. \citet{zhang2021multi} also reported the baseline based on the implementation of \citet{yu-etal-2020-improving-multimodal}, so we suspect the \textbf{UMGF} model cannot be improved as well.
Therefore, the under-trained textual baselines of previous work make the effectiveness of the images unclear and we show that some of the MNER models perform even weaker than our \textbf{BERT-CRF} model.

\begin{table}[t!]
\small
\centering
\setlength\tabcolsep{2pt}
\begin{tabular}{l||cc|cc|cc}
\hlineB{4}
\multicolumn{1}{c||}{} & \multicolumn{2}{c|}{Twitter-15} & \multicolumn{2}{c|}{Twitter-17}  & \multicolumn{2}{c}{SNAP} \\
 \hline
\multirow{3}{*}{Approach} & \multicolumn{2}{c|}{\bf Eval} & \multicolumn{2}{c|}{\bf Eval} & \multicolumn{2}{c}{\bf Eval}\\
 & \multicolumn{2}{c|}{\bf Modal} & \multicolumn{2}{c|}{\bf Modal} & \multicolumn{2}{c}{\bf Modal}\\
 & {\textbf{T}} & {\textbf{I+T}} & {\textbf{T}} & {\textbf{I+T}} & {\textbf{T}} & {\textbf{I+T}}\\
\hline\hline
{\textbf{ITA-Random}} & - & 74.67 & - & 84.98 & - & 85.82 \\
 {\textbf{ITA-GA$_{\text{BU}}$}} & - & 75.10 & - & 85.77 & - & 86.51 \\
{\textbf{ITA-LA$_{\text{BU}}$}} & - & 75.18 & - & 85.59  & - &  86.57\\
{\textbf{ITA-OCA$_{\text{Paddle}}$}} & - & 75.12 & - & \textbf{85.87}  & - & 86.66 \\
 {\textbf{BERT-CRF$_{\text{+ImgFeat}}$}} & - & 74.70 & - & 84.99 & - & 85.90\\
 {\textbf{VinVL-CRF}} & - & 60.58 & - & 75.55 & - & 74.53\\
 {\textbf{BERT+VinVL-CRF}} & - & 74.89 & - & 85.19 & - & 86.14 \\
 {\textbf{ITA-Joint}} & 74.88 & 75.22 & 85.31 & 85.60& 86.06 & 86.34  \\
\hline
 \multicolumn{7}{c}{\bf \textsc{References}}\\
\hline
 {\textbf{RpBERT w/o Rp}} & - & 72.60 & - & - & - & 86.20\\
 {\textbf{ITA-All$_{\text{+CVA}}$}} & \textbf{75.67} & \textbf{75.60} & \textbf{85.98} & 85.72 &  \textbf{86.83} & \textbf{86.75} \\
\hlineB{4}
\end{tabular}
\caption{A comparison of other variants of MNER models. }
\label{tab:comparison}
\end{table}

\subsection{Comparison with Other Variants}
To further show the effectiveness of ITA, we perform several comparisons between ITA and the following variants of the MNER model in Table \ref{tab:comparison}:
\paragraph{\textsc{\bf ITA-Random}:} We generate random image-text pairs for the model. For each sentence, we randomly select the image in the dataset and generate the corresponding visual contexts. The noises of random visual contexts make the model accuracy drop slightly comparing with our \textsc{\bf BERT-CRF} baseline, which shows the improvement of our approach is from the visual contexts rather than extending the input sequence length the embeddings.
\paragraph{\textsc{\bf ITA-Joint}:} It is an ablated model of \textsc{\bf ITA-All$_{\text{+CVA}}$}. We train the \textsc{\bf ITA-All} model for both input views without the CVA loss in Eq. \ref{eq:mda}. The model accuracy is improved moderately with only the \textbf{T} input view while our \textsc{\bf ITA-All$_{\text{+CVA}}$} can improve both input views significantly, which shows the effectiveness of the CVA module of ITA.
\paragraph{\textsc{\bf ITA-LA$_{\text{BU}}$} and \textsc{\bf ITA-GA$_{\text{BU}}$}:} We conduct experiments to see how the accuracy changes when using weaker image features. We use \textbf{B}ottom-\textbf{U}p features proposed by \citet{Anderson2017up-down} for object detection and image captioning. The captioning model is a pretrained image captioning model\footnote{\url{https://github.com/ruotianluo/self-critical.pytorch}} proposed by \citet{Luo2018DiscriminabilityOF} with the Bottom-Up features and self-critical training \citep{rennie2017self}. Results show that there is no significant difference between the visual contexts from Bottom-Up features and VinVL features. Therefore, our approaches can utilize other off-the-shelf vision models to extract visual contexts.
\paragraph{\textsc{\bf ITA-OCA$_{\text{Paddle}}$}:} We conduct experiments to see how the accuracy changes when using stronger OCR models. We use PaddleOCR\footnote{\url{https://github.com/PaddlePaddle/PaddleOCR}} for the experiment, which is one of the newest open resource lightweight OCR system. Results show that the model accuracy can be slightly improved comparing with \textsc{\bf ITA-OCA}, which shows the ITA models can be improved by using better OCR models.
\paragraph{{\textbf{BERT-CRF$_{\text{+ImgFeat}}$}}:} Instead of \textbf{ITA}, we can directly feed the image region features generated from an object detector into the BERT. We use ResNet-152 model to generate region features and then feed the features into a linear layer to project the region features into the same space of text features in the BERT. Moreover, we compare the model with {\textbf{RpBERT w/o Rp}}, which is an ablated model of {\textbf{RpBERT}} and is equivalent to {\textbf{BERT-CRF+$_{\text{+ImgFeat}}$}} over the usage of BERT embeddings. \citet{Sun2021RpBERTAT} showed {\textbf{RpBERT w/o Rp}} can improve the model accuracy compared with their baseline. However, our results show that the model accuracy slightly drops comparing with our {\textbf{BERT-CRF}}, which shows that it is difficult for the attention module of BERT to learn the relations of the unaligned representations of two modalities.
\paragraph{\textsc{\bf VinVL-CRF}:} To show how the pretrained V+L models perform on the NER task, we use VinVL since it is a very recent state-of-the-art pretrained V+L model on a lot of multi-modal tasks. We feed the VinVL model with texts and images in the MNER datasets and finetune the model over the task. We take the text representations output from VinVL as the input of the CRF layer. The accuracy of the finetuned VinVL model drops significantly compared to the BERT model, which shows that the inductive bias of the pretrained V+L model hurts the model accuracy on MNER.
\paragraph{\textsc{\bf BERT+VinVL-CRF}:} As the VinVL model may lead to an inductive bias over the common nouns and the image, we jointly finetune the BERT and VinVL models and concatenate the output text representations of the two models. The accuracy is improved on a moderate scale, which shows BERT is complementary to VinVL for MNER.

\subsection{Analysis}

\begin{filecontents}{twitter15.dat}
beam	acc
0	74.79
1	75.07
2	75.17
3	75.14
4	74.97
5	74.95
\end{filecontents}

\begin{filecontents}{twitter17.dat}
beam acc
0	85.18
1	85.61
2	85.75
3	85.58
4	85.41
5	85.27
\end{filecontents}

\begin{filecontents}{snap.dat}
beam acc
0   85.98
1	86.3
2	86.41
3	86.35
4	86.39
5	86.45

\end{filecontents}

\begin{figure}[t!]
\begin{minipage}{1.0\linewidth}
\centering
\begin{tikzpicture}

    \begin{axis}[
        xshift=-10cm,

        name=ner,

        width=0.42\textwidth,
        height=0.35\textwidth,

        legend columns=2,

        legend image post style={scale=0.5},
        legend style={font=\tiny,at={(0.9,-0.85)}, anchor=south east},
        tick label style={font=\scriptsize},
        xticklabels={0,1,5,10,15,20},
        xtick={0,1,2,3,4,5},
        ylabel style={font=\tiny,yshift=-0.2cm},
        ]
        \addplot[blue!80,mark=*] table[x=beam,y=acc] {twitter15.dat};
        \legend{Twitter-15}
    \end{axis}
    \begin{axis}[

        at={(ner.south west)},
        xshift=2.6cm,

        width=0.42\textwidth,
        height=0.35\textwidth,

        legend columns=2,

        legend image post style={scale=0.5},
        legend style={font=\tiny,at={(0.9,-0.85)}, anchor=south east},
        tick label style={font=\scriptsize},
        xticklabels={0,1,5,10,15,20},

        xtick={0,1,2,3,4,5},

        ylabel style={font=\tiny,yshift=-0.5cm},
        ]
        \addplot[red!80!yellow!45,mark=*] table[x=beam,y=acc] {twitter17.dat};
        \legend{Twitter-17}
    \end{axis}
    \begin{axis}[

        at={(ner.south west)},
        xshift=5.2cm,

        width=0.42\textwidth,
        height=0.35\textwidth,

        legend columns=2,

        legend image post style={scale=0.5},
        legend style={font=\tiny,at={(0.9,-0.85)}, anchor=south east},
        tick label style={font=\scriptsize},
        xticklabels={0,1,5,10,15,20},

        xtick={0,1,2,3,4,5},

        ylabel style={font=\tiny,yshift=-0.5cm},
        ymax=86.7,
        ymin=85.9,
        ]
        \addplot[yellow!80!blue!65,mark=*] table[x=beam,y=acc] {snap.dat};
        \legend{SNAP}
    \end{axis}
\end{tikzpicture}
\caption{A relation between the number of captions input to the MNER model and model accuracy. The x-axis is the number of captions. The y-axis is the averaged F1 score on the test set.}
\label{fig:test_curve}
\end{minipage}
\end{figure}

\paragraph{Effect of the Number of Captions}
Using more captions output from the captioning model can improve diversities of the visual contexts but can add noises to them as well. To better understand how the number of captions affects the model accuracy, we change the beam size and keep all the sentences output from the captioning model. The trends in Figure \ref{fig:test_curve} show that the model accuracy increases until $5$ captions for all the datasets and gradually drops when the number of captions further increases for Twitter-15 and 17 datasets. The observation shows that using $5$ captions keeps a good balance between the diversities and correctness of the captions.

\begin{filecontents}{mask_embed.dat}
type base ours ours-mda base_std ours_std ours-mda_std
Twitter-15 0.1848 0.3591 0.1581 0.0844 0.0324 0.0286
Twitter-17 0.3312 0.4725 0.1755 0.0879 0.0354 0.0112
SNAP 0.40 0.83 0.32 0.18 0.10 0.009
\end{filecontents}

\begin{figure}[t]
\centering
\begin{tikzpicture}

\begin{axis}[
    ybar=0pt,

    bar width = {1em},
    width=0.49\textwidth,
    height=0.18\textwidth,
    enlarge x limits={abs=1cm},
    symbolic x coords={Twitter-15,Twitter-17, SNAP},

    xticklabel style={

         align=center,
         font=\scriptsize,
      },
    yticklabel style={font=\small},
    ylabel = {L2 Distance},
    ylabel style = {yshift=-0.3cm,font=\small},
    legend style={font=\tiny,at={(0.5,-0.83)},anchor=south},

    legend columns=3,
    xtick={Twitter-15,Twitter-17, SNAP},
    legend image code/.code={%
      \draw[#1] (0cm,-0.1cm) rectangle (0.15cm,0.2cm);
    },
    ytick={0,0.5,1.0},
    ymin=0,
    ymax=1.0,
    ]
    \addplot[ybar, fill=blue!20, error bars/.cd,y dir=both,y explicit] table [x=type, y=base,y error=base_std] {mask_embed.dat};
    \addplot[ybar, fill=red!60!yellow!25, error bars/.cd,y dir=both,y explicit] table [x=type, y=ours,y error=ours_std] {mask_embed.dat};
    \addplot[ybar, fill=yellow!25, error bars/.cd,y dir=both,y explicit] table [x=type, y=ours-mda,y error=ours-mda_std] {mask_embed.dat};
    \legend{{\textbf{ BERT-CRF+ImgFeat}}, {\textbf{ITA-All}}, {\textbf{ITA-All$_{\text{+CVA}}$}}};
\end{axis}
\end{tikzpicture}
\caption{Averaged L2 distance between the token representations without image input (${\bm{r}}_i$) and with image input (${\bm{r}}^\prime_i$). The error bars mean the standard deviation over $5$ runs.}
\label{fig:l2dist}
\end{figure}

\paragraph{How ITA Eases the Cross-Modal Alignments}
Previous work such as \citet{moon-etal-2018-multimodal,Sun2021RpBERTAT} visualized modality attention in several cases to show the effectiveness of their approaches. However, visualizing the multi-layer attention in transformer-based embeddings is relatively difficult. Instead of studying special cases, we statistically calculate the averaged L2 distance between token representations ${\bm{r}}_i$ and ${\bm{r}}^{\prime}_i$ from two input modalities to show how the token representations depend on image information. In Figure \ref{fig:l2dist}, the L2 distance \textbf{ITA-All} is significantly larger than that of \textbf{BERT-CRF+ImgFeat}. Besides, the standard deviation of \textbf{BERT-CRF+ImgFeat} is very large. The observations show the image region features make the alignment become difficult and unstable while our visual contexts can significantly ease the cross-modal alignments. Moreover, with CVA, the L2 distance becomes much smaller and stable as CVA aligns the two input views to reduce the dependence on images, which shows the MNER model can better utilize the textual information with CVA.

\begin{table*}[t!]
\small
\centering
\begin{tabular}{l|ccc|ccc|ccc|ccc}
\hlineB{4}
 & \multicolumn{3}{c|}{LOC} & \multicolumn{3}{c|}{ORG} & \multicolumn{3}{c|}{PER} & \multicolumn{3}{c}{OTHER}\\% & \multicolumn{3}{c|}{LOC} \\% & \multicolumn{3}{c|}{ORG} & \multicolumn{3}{c|}{PER} & \multicolumn{3}{c}{OTHER}\\
 & P & R & F1 & P & R & F1 & P & R & F1 & P & R & F1\\
\hline\hline
 & \multicolumn{12}{c}{Twitter-15}\\
 \hline
{\textbf{BERT-CRF}} & 80.0 & 83.8 & 81.8 & 65.9 & 61.0 & 63.3 & 84.2 & 86.8 & 85.4 & 44.2 & 44.2 & 44.1\\
{{\textbf{ITA-All$_{\text{+CVA}}$}}} & 81.1 & 84.2 & 82.6 & 68.8 & 60.6 & 64.4 & 84.0 & 87.2 & 85.6 & 44.9 & 44.6 & 44.8 \\
$\mathbf{\Delta}$ & 1.1  & 0.4  & 0.8  & 2.8  & -0.4 & 1.1  & -0.2 & 0.4  & 0.1  & 0.8  & 0.5  & 0.6 \\
\hline\hline
 & \multicolumn{12}{c}{Twitter-17} \\
\hline
{\textbf{BERT-CRF}} & 85.5 & 84.4 & 84.9 & 83.5 & 83.8 & 83.7 & 90.7 & 90.8 & 90.7 & 68.9 & 65.1 & 66.9 \\
{{\textbf{ITA-All$_{\text{+CVA}}$}}} & 86.0 & 83.7 & 84.8 & 83.9 & 84.2 & 84.0 & 91.9 & 90.9 & 91.4 & 73.7 & 64.3 & 68.6 \\
$\mathbf{\Delta}$ & 0.5  & -0.7 & -0.1 & 0.3  & 0.4  & 0.4  & 1.2  & 0.1  & 0.7  & 4.8  & -0.8 & 1.7 \\
\hline\hline
 & \multicolumn{12}{c}{SNAP} \\
\hline
{\textbf{BERT-CRF}} & 82.1 &82.8 &82.5 &87.8 &86.9 &87.3 &91.0 &91.5 &91.2 &72.3 &75.1 &73.7 \\
{{\textbf{ITA-All$_{\text{+CVA}}$}}} & 80.3 &81.7 &81.0 &87.8 &86.5 &87.1 &90.1 &91.2 &90.6 &70.1 &73.2 &71.6\\
$\mathbf{\Delta}$ & 1.9 &1.1 &1.5 &0.6 &0.5 &0.5 &0.9 &0.3 &0.6 &2.2 &1.9 &2.1 \\
\hlineB{4}
\end{tabular}
\caption{A comparison between our ITA ({\textbf{ITA-All$_{\text{+CVA}}$}} with \textbf{I+T} inputs) model and the baseline ({\textbf{BERT-CRF}}) in precision (P), recall (R) and F1. $\mathbf{\Delta}$ represents the relevant improvement of ITA over the Baseline.}
\label{tab:prf}
\end{table*}

\paragraph{How Images Affect the NER Prediction}
To study the effectiveness of the images over each label, we show a comparison between our model and our baselines in Table \ref{tab:prf}. When the relative improvement of the F1 score is larger than $0.5$, the relative improvement of precision is larger than that of recall. The observation shows that the main improvement of MNER is mainly because the images can help the model to reduce false-positive predictions for disambiguation on uncertain entities.\footnote{In Appendix \ref{app:case}, we show several cases to show the effectiveness of ITA to affect NER prediction.}

\section{Related Work}

\paragraph{Multi-modal Named Entity Recognition}
Most of the previous approaches to MNER focus on the interaction between image and text features through attention mechanisms. \citet{moon-etal-2018-multimodal} proposed a modality attention network to fuse the text and image features before the input to the BiLSTM layer. \citet{lu-etal-2018-visual} additionally used a visual attention gate for the output features of the BiLSTM layer. \citet{zhang2018adaptive} proposed an adaptive co-attention network after the BiLSTM layer to model the interaction between image and text. Recently, \citet{10.1145/3394171.3413650} proposed OCSGA, which use object labels to model the interaction between image and object labels in an additional dense co-attention layer.
Compared with the work, we show a simpler and more effective way to utilize object labels and additionally use other alignment approaches to further improve the model accuracy. \citet{yu-etal-2020-improving-multimodal} proposed UMT, which utilized a multi-modal interaction module and an auxiliary entity span detection module for MNER. \citet{zhang2021multi} proposed UMGF, which utilizes a pretrained parser to create the graph connection between visual object tags and textual words. They used a graph attention network to fuse the textual and visual features. In order to better model whether the image is related to the text, \citet{Sun2021RpBERTAT} proposed RpBERT, which additionally trains on a text-image relation classification dataset proposed by \citet{vempala-preotiuc-pietro-2019-categorizing} to prevent the negative effect of noisy images. Comparing with RpBERT, we use CVA to let the NER model better utilize the input sentences without such kinds of supervision. All of these approaches focus on fusing the image and text features through the attention mechanism but ignore the gap between the image and text features while we propose to fully utilize the attention mechanism in the pretrained textual embeddings through aligning image features into textual space. Besides, some cross-media research also shows the effectiveness of OCR texts \citep{chen2016context,wang-etal-2020-cross-media} and object tags \citep{7780398} have been shown. Most of the approaches introduced a new attention module over cross-modal features while in comparison ITA effectively utilizes the attention module in the pretrained textual embeddings.

\paragraph{Pretrained Vision-Language Models}
Inspired by related work on language model pretraining, visual-language pretraining (VLP) has recently attracted a lot of attention \citep{li2019visualbert,Lu2019ViLBERTPT,chen2020uniter,Tan2019LXMERTLC,Li2020OscarOA,Yu2021ERNIEViLKE,zhang2021vinvl}. The pretrained V+L models are pretrained on large-scale image-text pairs and have achieved state-of-the-art accuracy over various vision-language tasks such as image captioning, VQA, NLVR and image-text retrieval.
Recently, \citet{Li2020OscarOA} proposed Oscar to add object tags in pretraining so that self-attention can learn the image-text alignments easily. Following Oscar, \citet{zhang2021vinvl} proposed VinVL to train a large-scale object detector to improve the pretrained V+L model's accuracy. Comparing with VLP, MNER is a totally different task.
Firstly, the image-caption pairs are given in VLP and the image and text are equally important in pretraining for general representations. Therefore, using global alignment is meaningless for VLP but makes sense for MNER. In MNER, the input text is not the caption of the image and the image may not adds additional information to the input text.
Secondly, though captions and object tags are often utilized in VLP, how to effectively utilize the captions and object tags of the image in MNER is rarely considered. Finally, besides the local and global alignments, another aspect of ITA is the optical character alignment and cross-view alignment, which is rarely considered in VLP.

\section{Conclusion}
In this paper, we propose Image-Text Alignments for multi-modal named entity recognition, which convert images into object labels, captions and OCR texts to align the image representations into textual space in a multi-level manner and form a cross-modal input view. The model can effectively utilize attention module of the transformer-based embeddings. Considering noises, availability of images and inference speed for practical use, we propose cross-view alignment, which let the MNER models better utilize the text information in the input. In our experiments, we show that ITA significantly outperforms previous state-of-the-art approaches on MNER datasets. We also show that most of the previous work failed to train a good textual baseline while our textual baseline can easily match or even outperform previous multi-modal approaches. In analysis, we further analyze how ITA eases the cross-modal alignments and how the images affect the NER prediction.

\section*{Acknowledgements}
This work was supported by the National Natural Science Foundation of China (61976139) and by Alibaba Group through Alibaba Innovative Research Program.


\section{Introduction}

Large LMs based on the Transformer architecture~\cite{NIPS2017_3f5ee243} have become increasingly popular since GPT~\cite{radford2018improving} and BERT~\cite{devlin-etal-2019-bert} were introduced, prompting the creation of many large LMs pretrained on English text \cite{NEURIPS2019_dc6a7e65,clark2020electra,NEURIPS2020_d6f1dd03,ram-etal-2021-shot}. There is a tendency towards training larger and larger models~\cite{brown2020language,fedus2021switch} while the main focus is on the English language. Recent work has called attention to the costs associated with training increasingly large LMs, including environmental and financial cost~\cite{strubell-etal-2019-energy, bender2021dangers}. If training large LMs for English is already costly, it is prohibitively expensive to train new, similarly powerful models to cover other languages.

One approach to address this issue is creating massively multilingual models
~\cite{devlin-etal-2019-bert,conneau-etal-2020-unsupervised,xue-etal-2021-mt5} trained on a concatenation of texts in many different languages. These models show strong natural language understanding capabilities in a wide variety of languages, but suffer from what \citet{conneau-etal-2020-unsupervised} call the \textit{curse of multilinguality}: beyond a certain number of languages, overall performance decreases on monolingual as well as cross-lingual tasks. Consistent with this finding, \citet{nozza2020mask} observe that monolingual LMs often outperform massively multilingual models. This might be attributed to superior quality of monolingual tokenizers over their multilingual counterparts~\cite{rust-etal-2021-good}. It is thus desirable to train monolingual models in more languages. Training monolingual models in non-English languages is commonly done by training a new model with randomly initialized parameters~\cite{antoun-etal-2020-arabert, louis2020belgpt2,martin-etal-2020-camembert,rekabsaz2019regularization}. However, to train a model with capabilities comparable to that of an English model in this way, presumably a similar amount of compute to what was used to train the English model would be required.

To address this issue, we introduce WECHSEL{},\footnote{Word Embeddings Can Help initialize Subword Embeddings in a new Language.} a novel method to transfer monolingual language models to a new language. WECHSEL{} uses multilingual static word embeddings between the source language and the target language to initialize model parameters. WECHSEL{} first copies all inner (non-embedding) parameters of the English model, and exchanges the tokenizer with a tokenizer for the target language. Next, in contrast to prior work doing random initialization~\cite{de-vries-nissim-2021-good}, the token embeddings in the target language are initialized such that they are close to semantically similar English tokens by mapping multilingual static word embeddings to subword embeddings. The latter step is particularly important considering that token embeddings take up roughly 31\% of the parameters of RoBERTa~\cite{liu2019roberta} and roughly 33\% of the parameters of GPT2~\cite{radford2019language}. Intuitively, semantically transferring embeddings instead of randomly initializing one third of the model should result in improved performance. Our parameter transfer provides an effective initialization in the target language, requiring significantly fewer training steps to reach high performance than training from scratch. As multilingual static word embeddings are available for many languages \cite{bojanowski2017enriching}, WECHSEL{} is widely applicable.

We conduct our experiments on RoBERTa and GPT-2 as representative models of encoder and decoder language models, respectively. We transfer the English RoBERTa model to four languages (French, German, Chinese and Swahili), and the English GPT-2 model to the same four plus another four very low-resource languages (Sundanese, Scottish Gaelic, Uyghur and Malagasy). We evaluate the transferred RoBERTa models on Named Entity Recognition (NER), and Natural Language Inference (NLI) tasks in the respective languages. The transferred GPT-2 models are evaluated in terms of Language Modelling Perplexity (PPL) on a held-out set. We compare WECHSEL{} with randomly initialized models (denoted as FullRand{}), as well as the recently proposed TransInner{} method which only transfers the inner (non-embedding) parameters~\cite{de-vries-nissim-2021-good}. All mentioned models are trained under the same conditions (around 4 days on a TPUv3-8). We also compare our model with models of comparable size trained from scratch under significantly larger training regimes, in particular CamemBERT~\cite{martin-etal-2020-camembert} (French), GBERT$_\mathrm{Base}$~\cite{chan-etal-2020-germans} (German), and BERT$_\mathrm{Base}$-Chinese~\cite{devlin-etal-2019-bert}.

Results show that models initialized with WECHSEL{} outperform randomly initialized models and models initialized with TransInner{} across all languages and all tasks, for both RoBERTa and GPT-2. In addition, strong performance is reached at a fraction of the training steps of other methods. Our contribution is summarized as follows.

\begin{itemize}
  \item We propose WECHSEL{}, a novel method for transferring monolingual language models to a new language by utilizing multilingual static word embeddings between the source and the target language.

  \item We show effective transfer of RoBERTa and GPT-2 using WECHSEL{} to four and eight languages, respectively, achieved after minimal training effort.

  \item We release more effective GPT-2 and RoBERTa models than previously published non-English models, achieved under our more efficient training setting. Our code and models are publicly available at \href{https://github.com/cpjku/wechsel}{\nolinkurl{github.com/cpjku/wechsel}}.

\end{itemize}

In the following, we review related work in Section~\ref{sec:related_work}. We introduce the WECHSEL{} method in Section~\ref{sec:methodology}, followed by explaining the experiment setup in Section~\ref{sec:experimental_design}. We show and discuss results in Section~\ref{sec:results}.

\section{Related Work}
\label{sec:related_work}

\paragraph{Large Language Models.} Training Language Models is usually done in a self-supervised manner i.\,e.  deriving labels from the training text instead of needing explicit annotations. One optimization objective is Masked Language Modelling \cite[MLM]{devlin-etal-2019-bert}, where randomly selected tokens in the input are replaced by a special \verb|[MASK]| token, and the task is to predict the original tokens. Another common objective is Causal Language Modelling (CLM), where the task is to predict the next token. These two objectives highlight a fundamental distinction between language models: models can be trained as encoders (e.g. with MLM) or as decoders (e.g. with CLM).

Instead of words, the vocabulary of recently proposed language models commonly consists of subwords~\cite{clark2020electra, liu2019roberta, devlin-etal-2019-bert}.

\paragraph{Multilingual representations.}

There has been a significant amount of work in creating multilingual static word embeddings. A common method is learning embeddings from scratch using data in multiple languages~\cite{luong-etal-2015-bilingual,duong-etal-2016-learning}.
Alternatively, multilinguality can be achieved by aligning existing monolingual word embeddings using a bilingual dictionary, so that the resulting embeddings share the same semantic space~\cite{xing-etal-2015-normalized,joulin-etal-2018-loss}. Recent studies improve on this by reducing (or completely removing) the need for bilingual data~\cite{artetxe-etal-2017-learning,artetxe-etal-2018-robust,lample2018word}.

Beside static word embeddings, multilinguality is also well studied in the area of contextualized representations. One approach to learn multilingual contextualized representations is through training a model on a concatenation of corpora in different languages. Some models created based on this approach are mBERT~\cite{devlin-etal-2019-bert}, XLM-R~\cite{conneau-etal-2020-unsupervised} and mT5~\cite{xue-etal-2021-mt5}, trained on text in 104, 100, and 101 languages, respectively. As shown by \citet{pires-etal-2019-multilingual}, a multilingual model such as mBERT can enable cross-lingual transfer by using task-specific annotations in one language to fine-tune the model for evaluation in another language. Despite the benefits, recent studies outline a number of limitations of massively multilingual LMs. \citet{wu-dredze-2020-languages} empirically show that in mBERT
``the 30\% languages with least pretraining resources perform worse than using no pretrained language model at all''. \citet{conneau-etal-2020-unsupervised} report that beyond a certain number of languages in the training data, the overall performance decreases on monolingual as well as cross-lingual tasks. These studies motivate our work on introducing an efficient approach for creating effective monolingual LMs for more languages.

\paragraph{Cross-lingual transfer of monolingual LMs.} Studies in this area can be divided into two categories:

\begin{itemize}
    \item \textbf{Bilingualization of a monolingual LM} is concerned with extending a model to a new language while preserving its capabilities in the original language. \citet{artetxe-etal-2020-cross} approach this problem by replacing the tokenizer and relearning the subword embeddings, while freezing other (non-embedding) parameters. Such a model becomes bilingual, since the initial tokenizer and embeddings can be used for tasks in the source language, while the new tokenizer and embeddings can be used for tasks in the target language. Thus, a model can be finetuned on annotated task data in the source language, and then zero-shot transferred to the target language. \citet{tran2020english} follow a similar approach, while instead of randomly initializing embeddings, they utilize static word embeddings to initialize embeddings in the target language close to semantically similar English tokens. They then continue training the model on an English text corpus as well as on the target language in order to preserve model capabilities in English.

    \item \textbf{Creating a new monolingual LM in the target language} is, in contrast, concerned with transferring a model from a source to a target language without the necessity to preserve its capabilities in the source language. \citet{zoph-etal-2016-transfer} and \citet{nguyen-chiang-2017-transfer} show that cross-lingually transferring a machine translation model can improve performance, especially for low-resource languages. \citet{zoph-etal-2016-transfer} use embeddings of random tokens in the original vocabulary to initialize token embeddings in the new vocabulary, while \citet{nguyen-chiang-2017-transfer} utilize vocabulary overlap between the source and target language. More recently, \citet{de-vries-nissim-2021-good} follow a similar approach to the one of \citet{artetxe-etal-2020-cross} for transferring a GPT-2 model to a new language. \citet{de-vries-nissim-2021-good} add an additional step, where they train the entire model for some amount of steps to allow adapting to the target language beyond the lexical level. We refer to the method of \citet{de-vries-nissim-2021-good} as TransInner{} and consider it as a baseline in our experiments.

\end{itemize}

Our WECHSEL{} method belongs to the second category. WECHSEL{} can be seen as an extension to the method proposed by \citet{tran2020english} with the goal of creating a new monolingual LM instead of bilingualizing the LM. This allows removing the constraints imposed by the need to preserve the model's capabilities in the source language. In addition, we generalize the semantic subword mapping done by \citet{tran2020english} to consider an arbitrary number of semantically similar subword with an arbitrary temperature. We are the first to show that a cross-lingually transferred model can outperform monolingual models which have been trained extensively from scratch in the target language, while requiring substantially less computational resources.

\section{Methodology}
\label{sec:methodology}

To initialize the model in the target language, we copy the inner (non-embedding) parameters from the source model. Our goal, then, is given the tokenizer $T^s$ in the source language with vocabulary ${\mathbb{U}}^s$, the corresponding token embeddings ${\bm{E}}^s$, and a tokenizer $T^t$ in the target language with vocabulary ${\mathbb{U}}^t$, to find a good initialization of the embeddings ${\bm{E}}^t$ by using ${\bm{E}}^s$. To this end, we use existing bilingual word embeddings enriched with subword information, containing a set of words and subword n-grams in the source and target language and their aligned vectors. We denote the set of words and n-grams in the source and target language as $\sV$$^{s}$ and $\sV$$^{t}$ respectively, and the aligned static embeddings as \wordembeddings{s} and \wordembeddings{t}. In Appendix~\ref{appendix:no_subword_information} we consider an alternative method if no subword information is available in the bilingual word embeddings.

First, independently for both languages, we compute static subword embeddings for tokens in the tokenizer vocabulary in the same semantic space as the static word embeddings (Section \ref{section:subword-embedding-computation}). This results in subword embeddings \subwordembeddings{s} and \subwordembeddings{t} for the source and target language, respectively. Next, we use \subwordembeddings{s} and \subwordembeddings{t} to compute the semantic similarity of every subword in ${\mathbb{U}}^s$ to every subword in ${\mathbb{U}}^t$. Using these semantic similarities, we initialize the embeddings in ${\bm{E}}^t$ through a convex combination of embeddings in ${\bm{E}}^s$ (Section \ref{section:similarity-based}). By applying WECHSEL, the vectors of ${\bm{E}}^t$ are in the same semantic space as ${\bm{E}}^s$, where a subword in the target language is semantically similar to its counterpart(s) in the source language. These steps are summarized in Figure \ref{fig:flow} and explained in more detail in the following.

\begin{figure}[t]
\hspace*{-0.15cm}\includegraphics[width=7.8cm]{flow}
\caption{Summary of our WECHSEL{} method. We show \textcolor{blue}{inputs}, \textcolor{orange}{intermediate results} and \textcolor{teal}{outputs}.}
\centering
\label{fig:flow}
\end{figure}

\subsection{Subword Embedding Computation}
\label{section:subword-embedding-computation}

The process of mapping word embeddings to subword embeddings is done individually for the source and the target language. Given a tokenizer $T$ with vocabulary ${\mathbb{U}}$ and embeddings \wordembeddings{}, the goal is to find subword embeddings \subwordembeddings{} for subwords in ${\mathbb{U}}$ in the same semantic space as \wordembeddings{}. To this end, we decompose subwords in ${\mathbb{U}}$ into n-grams and compute the embedding by taking the sum of the embeddings of all occuring n-grams, equivalent to how embeddings for out-of-vocabulary words are computed in fastText~\cite{bojanowski2017enriching}.

\[
{\bm{u}}_{x} = \sum_{g \in {\mathbb{G}}^{(x)}} {\bm{w}}_{g}
\]
where ${\mathbb{G}}^{(x)}$ is the set of n-grams occuring in the subword $x$ and ${\bm{w}}_{g}$ is the embedding of the n-gram~$g$. Subwords in which no known n-gram occurs are initialized to zero.

\subsection{Subword similarity-based Transfer}
\label{section:similarity-based}
Applying the previous step to both source and target language results in the subword embeddings \subwordembeddings{s} and \subwordembeddings{t} over the subword vocabularies ${\mathbb{U}}^s$ and ${\mathbb{U}}^t$, respectively. Our aim is to leverage these embeddings to find an effective transformation from ${\bm{E}}^s$ to ${\bm{E}}^t$. We first compute the cosine similarity of every subword $x \in {\mathbb{U}}^t$ to every subword $y \in {\mathbb{U}}^s$, denoted as $s_{x,y}$.
\[
s_{x,y} = \frac{{\bm{u}}^{t}_{x} {{\bm{u}}^{s}_{y}}^T}{\|{\bm{u}}^{t}_{x}\|\|{\bm{u}}^{s}_{y}\|}
\]

We now exploit these similarities to initialize embeddings in ${\bm{E}}^t$ by a convex combination of embeddings in ${\bm{E}}^s$. In particular, each subword embedding in ${\bm{E}}^t$ is defined as the weighted mean of the $k$ nearest embeddings in ${\bm{E}}^s$ according to the similarity values. The weighting is done by a softmax of the similarities with temperature $\tau$.
\[
{\bm{e}}_{x}^{t} = \frac{\sum_{y \in \mathcal{J}_x} {\exp{(s_{x,y}/\tau)} \cdot {\bm{e}}_{y}^{s}}}{\sum_{y' \in \mathcal{J}_x} \exp{(s_{x,y'}/\tau)}}
\]
where $\mathcal{J}_x$ is the set of $k$ neighbouring subwords in the source language. Subword embeddings for which \subwordembeddings{t} is zero are initialized from a random normal distribution $\mathcal{N}(\mathbb{E}[{\bm{E}}^s], \mathrm{Var}[{\bm{E}}^s])$.

\section{Experiment Design}
\label{sec:experimental_design}

We evaluate our method by transferring the English RoBERTa~\cite{liu2019roberta} and the English GPT-2 model~\cite{radford2019language} to French, German, Chinese and Swahili. We refer to these languages as \textit{medium-resource languages}. In addition, we study the benefits of our method on four \textit{low-resource languages}, namely Sundanese, Scottish Gaelic, Uyghur and Malagasy.

We evaluate WECHSEL-RoBERTa{} by fine-tuning on XNLI \cite{conneau2018xnli}, and on the balanced train-dev-test split of WikiANN \cite{rahimi-etal-2019-massively, pan-etal-2017-cross} to evaluate NLI and NER performance, respectively. The hyperparameters used for fine-tuning are reported in Appendix~\ref{appendix:hyperparameters}. GPT-2 is evaluated by Perplexity (PPL) on a held-out set from the same corpus on which the model was trained on. Due to the difficulty of extrinsic evaluation on low-resource languages, we only train GPT-2 models in these languages, and evaluate their performance intrinsically via Language Modelling Perplexity on a held-out set.
We use the pretrained models $\textrm{RoBERTa}_{\mathrm{Base}}$ with 125M parameters, and the small GPT-2 variant with 117M parameters provided by HuggingFace's Transformers~\cite{wolf-etal-2020-transformers} in all experiments.

Since under limited training regimes such as ours, using a smaller corpus does not in general degrade performance~\cite{martin-etal-2020-camembert}, we use a subset of 4GiB from the OSCAR corpus for German, French and Chinese.
For the other languages, we use data from the CC-100 corpus \cite{conneau-etal-2020-unsupervised} which contains 1.6GiB, 0.1GiB, 0.1GiB, 0.4GiB and 0.2GiB for Swahili, Sundanese, Scottish Gaelic, Uyghur and Malagasy, respectively. To obtain aligned word embeddings between the source and the target language we use monolingual fastText word embeddings\footnote{https://fasttext.cc}
~\cite{bojanowski2017enriching}. We align these embeddings using the Orthogonal Procrustes method~\cite{schonemann1966generalized,artetxe-etal-2016-learning} with bilingual dictionaries from MUSE\footnote{https://github.com/facebookresearch/MUSE}~\cite{conneau2017word} for French, German and Chinese and a bilingual dictionary from FreeDict\footnote{https://freedict.org}~\cite{banski2009freedict} for Swahili. For the low-resource languages, we use bilingual dictionaries scraped from Wiktionary.\footnote{available at \href{https://github.com/cpjku/wechsel}{\nolinkurl{github.com/cpjku/wechsel}}}

We choose temperature $\tau=0.1$ and neighbors $k=10$ for WECHSEL{} by conducting a parameter search over a grid with varying values for $k$ and $\tau$ using linear probes (Appendix~\ref{appendix:grid_search}). We train tokenizers in the target languages using a vocabulary size of 50k tokens and byte-level BPE~\cite{radford2019language}. After applying WECHSEL{}, we continue training RoBERTa on the MLM objective and GPT-2 on the CLM objective. We compare against two baseline methods.
 \begin{itemize}
     \item \textbf{TransInner:} Randomly initializing ${\bm{E}}^t$ while transferring all other parameters from the English model as in \citet{de-vries-nissim-2021-good}. After training only embeddings for a fixed amount of steps while freezing other parameters, the entire model is trained for the remaining steps. In preliminary experiments reported in Appendix~\ref{appendix:choosing_baseline}, we compare the method by \citet{zoph-etal-2016-transfer} with TransInner{}, observing superior performance of TransInner{}, so we choose TransInner{} as the baseline for cross-lingual transfer in all our experiments.

     \item \textbf{FullRand:} Training from scratch in the target language, as is commonly done when training BERT-like or GPT-like models in a new language \cite{antoun-etal-2020-arabert, louis2020belgpt2, chan-etal-2020-germans, martin-etal-2020-camembert}.
 \end{itemize}

\input{tables/training_time}

\input{tables/roberta_results}

All models are trained for 250k steps with the same hyperparameters across all languages (reported in Appendix \ref{appendix:hyperparameters}). Training one model takes around 4 days on a TPUv3-8. For WECHSEL{} and FullRand{} we use a learning rate (LR) schedule with linear warmup from zero to the peak LR for the first 10\% of steps, followed by a linear decay to zero. For TransInner{}, we perform two warmup phases from zero to peak LR, once for the first 10\% of steps for training embeddings only, then again for the remaining steps while training the entire model.

In addition to the mentioned baselines trained under this setting, we compare the results of RoBERTa models with existing comparable models trained from scratch with more training effort. We consider the total number of tokens the model has encountered in the target language, computed as the product of batch size $\times$ sequence length $\times$ train steps (shown in Table \ref{table:training_time}) as a proxy for training effort. We evaluate the performance of CamemBERT~\cite{martin-etal-2020-camembert} (French), GBERT$_{\mathrm{Base}}$~\cite{chan-etal-2020-germans} (German), and BERT$_\mathrm{Base}$-Chinese~\cite{devlin-etal-2019-bert} as existing monolingual LMs,\footnote{To the best of our knowledge there is no monolingual model available for Swahili.} as well as XLM-R$_{\mathrm{Base}}$~\cite{artetxe-etal-2020-cross} as a high-performing multilingual LM.

\section{Results}
\label{sec:results}

We present our results on transferring RoBERTa and GPT-2 from English to other languages, followed by analyzing training behavior. In Appendix~\ref{appendix:closest_tokens}, we provide a qualitative assessment of how well subword tokens are mapped between the source and the target languages.

\subsection{Transferring RoBERTa}

Table~\ref{table:roberta_results} reports the evaluation results of RoBERTa. As shown, models initialized with WECHSEL{} outperform models trained from scratch and models initialized with TransInner{} across all languages. Surprisingly, close relatedness of the source and target language is not necessary to achieve effective transfer, as e.\,g.  on NLI WECHSEL{} improves absolute accuracy by $7.15\%$, $6.31\%$, $6.94\%$ and $4.71\%$ over models trained from scratch for French, German, Chinese and Swahili, respectively.

\begin{figure*}[t]
\hspace*{-0.5cm}\includegraphics[width=16.5cm]{scores}
\caption{Test scores over training steps from fine-tuning RoBERTa models on NLI (using XNLI) and NER (using WikiANN). Perplexity on the held-out set over training steps of GPT-2 models. We evaluate every 12.5k steps.}
\centering
\label{fig:scores}
\end{figure*}

We observe that our parameter transfer-based model consistently outperforms the previously released LMs on both monolingual and multilingual settings, while these models benefit from much larger training resources in terms of computation time and corpus size. In particular, the results show an improvement over XLM-R$_{\mathrm{Base}}$ by an average 3.54\% accuracy for NLI and 1.14\% micro F1 score for NER. For NLI, we improve over the prior monolingual models by $1.55\%$, $3.15\%$ and $1.77\%$ absolute accuracy for French, German and Chinese, respectively. For NER, we observe improvements over monolingual models with $0.62\%$ and $0.26\%$ absolute micro F1 score improvement for French and German, respectively. For Chinese, the monolingual model BERT$_\mathrm{Base}$-Chinese still outperforms our method by $1.5\%$ absolute micro F1 score. We suspect that the discrepancy between NLI and NER is due to the limited training corpus size (max. 4GiB), while a larger corpus can potentially improve NER as more named entities appear~\cite{martin-etal-2020-camembert}.

\input{tables/gpt2_results}

\input{tables/low_resource_results}

The first two columns of Figure~\ref{fig:scores} show the performance of RoBERTa models on downstream tasks after each 12.5k training steps. Models initialized with WECHSEL{} reach high performance in significantly fewer steps than models initialized with FullRand{} or TransInner{}.

We expect FullRand-RoBERTa{} to approach performance of the respective prior monolingual models when trained on the same amount of tokens.\footnote{It would presumably be slightly worse because we restrict training corpus size to 4GiB.}
For French, WECHSEL-RoBERTa{} outperforms CamemBERT after 10\% of training steps, reducing training effort by 64x. For German, WECHSEL-RoBERTa{} outperforms GBERT$_\mathrm{Base}$ after 10\% of training steps, reducing training effort by 39x. For Chinese, WECHSEL-RoBERTa{} outperforms BERT$_\mathrm{Base}$-Chinese on NLI, but does not
outperform BERT$_\mathrm{Base}$-Chinese on NER.

\subsection{Transferring GPT-2}

\subsubsection{To Medium-Resource Languages}

Results on medium-resource languages are shown in Table~\ref{table:gpt_results}. Similar to the results for WECHSEL-RoBERTa{}, the GPT-2 models trained with WECHSEL{} consistently outperform the models trained from scratch and with TransInner{} across all languages.

The rightmost column of Figure~\ref{fig:scores} depicts the performance of GPT-2 models after each 12.5k training steps. Comparing the results across all languages throughout training, we observe a stronger dependence on similarity of the source to the target language than for downstream tasks such as NLI or NER. In particular, for French and German, WECHSEL{} is consistently better than TransInner{} and FullRand{} throughout the entire training, while for Chinese, a decrease in perplexity towards the end of training causes WECHSEL{} to surpass training from scratch.

\begin{figure}[h]
\includegraphics[width=7.9cm]{low_resource_scores}
\caption{Perplexity throughout training on low-resource languages. We evaluate every 2.5k steps and stop training if Perplexity on the held-out set does not improve for 10k steps.}
\centering
\label{fig:low_resource_scores}
\end{figure}

\subsubsection{To Low-Resource Languages}
Table~\ref{table:low_resource_results} reports the perplexity of Language Modelling on the low-resource languages. Again, we observe consistent improvements using WECHSEL{} on all languages. Furthermore, we find that the improvement from WECHSEL{} tends to increase as the amount of training data decreases by conducting a sensitivity analysis w.\,r.\,t.  the amount of available training data (Appendix \ref{appendix:ablation}).

In Figure \ref{fig:low_resource_scores} we report the performance of the low-resource LMs on the held-out set throughout training. One difference of the low-resource models with the ones trained on medium-resource languages is that the low-resource LMs are prone to overfitting, and require appropriate model selection even in the early steps of training. Notably, TransInner-GPT2{} takes more steps to overfit since all non-embedding parameters are frozen for the first 25k steps (c.\,f.  Section \ref{sec:experimental_design}).

\subsection{Is freezing necessary?}
\label{sec:freezing}

Previous work using the TransInner{} method freezes non-embedding parameters for a fixed amount of steps before training the entire model~\cite{de-vries-nissim-2021-good}. This is done to prevent catastrophic forgetting at the beginning of training. To evaluate if freezing non-embedding parameters is still necessary with our method, we conduct an additional experiment. We train a German GPT-2 model with WECHSEL{} and a model with TransInner{} without freezing any parameters, and the same models with freezing of non-embedding parameters for the first 10\% of steps. We match hyperparameters of the main experiments except training for 75k steps only. Based on the results shown in Figure \ref{fig:freezing}, we conclude that freezing is necessary when using TransInner{}, but there is no need for freezing when using WECHSEL{}

\begin{figure}[t]
\vspace*{-0.2cm}
\includegraphics[width=8cm]{freezing}
\caption{Comparison of German GPT-2 models trained with WECHSEL{} and TransInner{} between freezing non-embedding parameters at the start and not freezing any parameters.}
\centering
\label{fig:freezing}
\vspace*{-0.4cm}
\end{figure}

\section{Limitations and Potential Risks}

\vspace{-0.1cm}
\subsection{Limitations}

We conduct our experiments on up to eight languages, showing the benefits of our parameter transfer method to both medium- and low-resource languages. However, there are many more languages with diverse linguistic characteristics on which our WECHSEL{} method is not tested. This is a limitation forced by computational constraints, as we can not ascertain whether transfer to all other languages would result in similar improvements. In addition, our extrinsic evaluation is limited to two tasks (NLI and NER). While this choice is due to the limitations on the available collections in various languages, this evaluation does not necessarily provide a comprehensive view of language understanding tasks.

\vspace{-0.1cm}
\subsection{Risks}

It is well-known that existing LMs trained on English text encode societal biases~\cite{bolukbasi2016man,caliskan2017semantics,Rekabsaz0HH21}
 and stereotypes and using them in downstream tasks might lead to unfair treatment of various social groups~\cite{zerveas2022mitigating,krieg2022do,ganhoer2022mitigating,rekabsaz2021societal,melchiorre2021investigating,rekabsaz2020neural,elazar2018adversarial}. Since we propose a method to transfer the English LMs to new languages, it is highly probable that the existing biases are also transferred to the target LMs. We therefore advocate a conscious and responsible use of the transferred LMs in practice.

\section{Conclusion}

We introduce WECHSEL{}, an effective method to transfer monolingual language models to new languages. WECHSEL{} exploits multilingual static word embeddings to compute an effective initialization of subword embeddings in the target language. We conduct experiments by transferring RoBERTa and GPT-2 models from English to French, German, Chinese and Swahili, as well as English GPT-2 to four low-resource languages. The evaluation results show that the transferred RoBERTa and GPT-2 models are more efficient and effective than strong baselines, and consistently outperform prior monolingual models that have been trained for a significantly longer time. WECHSEL{} facilitates the creation of effective monolingual LMs for new languages with medium to low resources, particularly in computationally-limited settings. In addition, our work provides strong evidence towards the hypothesis by \citet{artetxe-etal-2020-cross} that deep monolingual language models learn abstractions that generalize across languages.

\section{Acknowledgments}

Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC). We thank Andy Koh and Artus Krohn-Grimberghe for providing additional computational resources. The ELLIS Unit Linz, the LIT AI Lab, the Institute for Machine Learning, are supported by the Federal State Upper Austria. We thank the project INCONTROL-RL (FFG-881064). Research also supported in part by the NSF (IIS-1956221), the State of Upper Austria and the Austria's Federal Ministry of Education, Science, and Research through the project FAIRFLOW (LIT-2021-YOU-215).