\section{Introduction}
 The Agebots project "Implementation of service robotics in the field of elderly care", is an interdisciplinary, federally funded project following a human-centered and iterative approach to support the elderly care with service robotic solutions. It focuses on the needs and challenges of the caregivers and the elderly in these care facilities. Different design thinking methods are used to elicit those needs and challenges, including multiple interviews with potential end-users, observations, shadowings, personae, rapid prototyping, laboratory, and field testing. Considering the different dimensions, i.e., the human, technology, business, and law aspects, prevents a one-sided, technical driven approach and promotes the implementation of requirement-based solutions.
The increase in the elderly population in Switzerland will reach an expected share of over 27\% of the 65+ population group (2.8 million) by 2050. This will cause major challenges, especially for retirement and nursing homes \cite{fur2017bevolkerung, suisse2015zukunft}. This ageing of society, in combination with work overload, changing care needs, lack of time, human errors, and economic pressure lead to a decline in the quality of care. To counteract this decline and ensure the satisfaction of the elderly and caregivers, the care facilities are dependent on innovative and financially feasible solutions \cite{klein2018robotik}. These innovative solutions should ensure a high-quality, and efficient care for the elderly in the future. Furthermore, the solutions should relieve the caregivers to give them more time for interpersonal activities. Considering these developments, service robotics might be the solution.

\section{The Agebots Project}
The Agebots project is structured in such a way that the robotic solutions are tailored to the challenges and needs of the end-users and are developed and tested in two iterations (cf. Table \ref{table_1}). The focus of the first testing was the technical feasibility and implementation. Here the robotic solutions were tested first with non-users. In the second Lab Test the robot systems were tested with end-users, i.e., elderly and caregivers of the retirement and nursing home. The focus of the second testing was on acceptance in terms of user-friendliness, readiness to use and personal impression. For each of the Lab Tests, a concept was created with specific observation criteria. The application fields could be tested several times by different users resulting in a variety of feedbacks and inputs. In addition, observations were carried out which allowed for identification of interaction difficulties with the robots. The feedback was implemented onsite (laboratory), allowing to continue the testing with the improved functionalities. Specifically, this iterative approach helped to implement and test the end-users' requirements directly, so that their wishes could be considered in the best possible way. The range of applications, which were tested in the two Lab Tests, were reduced to the most feasible or most promising ones throughout the testing phases.
\begin{table}[h]
\caption{Project Phases and general Results}
\label{table_1}
\begin{center}
\begin{tabular}{|p{2cm}|p{6cm}|}
\hline
Project phases & Results\\
\hline
\hline
Understanding & One hundred forty-four challenges for the caregivers and the elderly were identified.\\
\hline
Ideation & Fifteen application fields with 170 ideas for caregivers and ten application fields with 88 ideas for elderly were generated.\\
\hline
Technology & Seven robotic systems were procured, and functional tests were conducted.\\
\hline
Lab test 1 & Feasibility tests carried out with non-users (students, employees from the retirement and nursing home, and the universities), and first legal analysis was conducted. 25 applications were reduced to 12.\\
\hline
Iteration 1 & Application development of the robot systems based on findings of lab test 1. Research framework for the evaluation of the qualitative and quantitative data using a relational database was created and filled with insights.\\
\hline
Lab test 2 & Usability and acceptance tests with the caregivers and elderly of the specific retirement and nursing home in a laboratory setting. A business plan for the development of new markets for the implementation partner (here robot manufacturer) was elaborated.\\
\hline
Field testing (planned) & Testing of legally compliant, desirable, feasible, and economical application in real-world environments in the retirement and nursing home.\\
\hline
Marketing (planned) & "Sale" and distribution of the service robot systems in German-speaking countries by the main implementation partner (robot manufacturer).\\
\hline
\end{tabular}
\end{center}
\end{table}

\section{Companion Challenge}
One of the 144 identified challenges was the Companion Challenge which will now be discussed. Caregivers need to escort the elderlies several times a day to different locations within the retirement and nursing homes. A necessary but very time consuming activity for the caregivers, identified in the literature as the most requested feature to maintain the independence of the elderly \cite{hebesberger2015staff, pineau2003towards}. Therefore, the goal of this project is to find innovative assistance systems and robotic solutions, which can relieve or support the caregivers in this task, and preserve the elderlies independence.

\section{The Autonomous Rollator Design Process}
In this section we will discuss three iterations of the development of our autonomous rollator as an escorter robot.
\subsection{First iteration}
In the case of the Companion Challenge, one result of a creativity workshop was a rollator combined with a wheelchair. This first low-fidelity prototype was made of cardboard, cord, post-its and drawings (cf. Figure \ref{fig:experiment_1}).
At this workshop, the technical requirements for a possible solution were discussed and defined with the end-users, which were then procured, provided, and put into operation.

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{RollatorV02.jpg}

  \caption{Low-fidelity prototype of the rollator}
 \label{fig:experiment_1}
\end{figure}

\subsection{Second iteration}
In the first Lab Test (see Table \ref{table_1}), the robot systems and the evaluated application were tested with different non-users such as students and employees of universities as well as employees of the retirement and nursing home.

During the technical development, the following requirements for the Companion System were identified in accordance with the findings of \cite{hawes2017strands,pineau2003towards}:
\begin{itemize}
\item[a)]	Determining the position and orientation in space
\item[b)]	Generating a map, annotating important locations, and restricting areas
\item[c)]	Path planning considering temporary obstacles and physical variability
\item[d)]	Ensuring long-term autonomy of a system
\item[e)]	Determining the current floor
\item[f)]	Communicating with, monitoring of, and adapting to the user
\item[g)]	Ease of operation
\end{itemize}
Based on these technical requirements we adjusted the rollator from ello to escort the elderly autonomously.
In the following, we will discuss two iterations of modification of this rollator to become an autonomous robot.

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{RollatorV1.png}

  \caption{(a) Picture of the prototype in the first iteration. (b) Graphical user interface example used in the first iteration.}
 \label{fig:experiment_2}
\end{figure}
As a second iteration, we used the Wizard of Oz (WOZ) approach \cite{dahlback1993wizard}. The aim was to get an initial feeling for the requirement f) and the associated needs of the system. For this purpose, a tablet was mounted on a rollator. By means of the tablet, the wizard interacted visually and acoustically with the elderly (cf. Figure \ref{fig:experiment_2}).

The results achieved with the second prototype proved that communication via tablet works in principle. The WOZ approach allowed the wizard to interact with and actively influence the behavior of the elderly. Though, implementing this kind of interaction and influence on a computer and realizing an autonomous accompanying system with it is challenging. Additionally, it is difficult for the elderly to concentrate on the path and read the information on the tablet at the same time. This is especially problematic for people who are no longer sufficiently mobile. In summary, a communication tool other than the tablet still needs to be found.

\subsection{Third iteration}
For the second Lab Test (cf. Table \ref{table_1}), the rollator was rebuilt, as can be seen in Figure \ref{fig:experiment_3}, to tackle the previously mentioned requirements a) to c). The aim was to shift from the WOZ approach to an autonomous system and to become better acquainted with the possible applications. The rollator was equipped with two RLS RMB14 encoders connected to a BeagleBone blue single-board computer and a RPLIDAR A1 light detection and ranging sensor connected to a laptop. The BeagleBone blue and the laptop communicate over TCP/IP. Using odometry the position and orientation of the rollator is estimated. These calculations are done in the EEROS framework, and the results are sent to the ROS2 software stack \cite{thomas2014next}. To perform simultaneous localization and mapping (SLAM) the slam toolbox \cite{macenski2019use,macenski2021slam}, and for the navigation the Nav2 project \cite{macenski2020marathon} is used.
\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{RollatorV2.png}

  \caption{(a) Picture of the prototype of the third experiment. (b) System overview of the third experiment.}
 \label{fig:experiment_3}
\end{figure}
Caregivers were able to test the third prototype in a replicated elderly residence. Their feedback was positive, and they see potential in the application. From a technical point of view, the increasing computational effort required by SLAM with the size of the mapped area is a limiting factor. Another problem was that the rollator localization sometimes suddenly switched its location to the other side of a wall when it came too close to the wall. However, this was only observed in the replicated retirement and nursing home and may be understood as the walls are made of thin wooden panels. Nevertheless, this caused the map to become unusable and therefore the application had to be restarted. A possible solution would be to switch to a static map and only use SLAM for the initial recording of the map. Overall, further research is required to improve the prototype and tackle the remaining requirements.
\section{Discussion and Conclusion}
The human-centered approach in the analysis and implementation phase of technical assistance systems offers the possibility to align solutions directly with the needs of end-users. This allows corrections in the design process before the robotic solutions are implemented in the real environment. Through repeated iterative testing, prototypes are continuously adapted and improved.

Well-defined requirements for a companion system were identified and three prototypes were successfully developed to gain experience in dealing with these challenges. Problems in communicating with the elderly via a display were identified using the WOZ approach. The technical feasibility of localisation and navigation with a walker was demonstrated and the solution was evaluated by caregivers.

The results obtained in the second iteration showed that further research is needed in the identified challenges f) communicating with, monitoring of, and adapting to the user, as well as g) ease of operation. One possible solution, which will be investigated in a follow up study, is the implementation of force guided navigation. Like a lane departure warning system, the aim is to subtly point the user in the right direction. Ultimately leading to an intuitive communication that leaves the user's visual senses free for other tasks.

Although some software improvements are still needed to ensure the long-term autonomy, the prototype built in the third iteration demonstrated that escorting is possible with simple and inexpensive sensors. Furthermore, the positive oral feedback from caregivers about the usefulness of such a system is an important factor in deciding that the system should be further developed in a subsequent study.

Two steps are planned for the further investigation. In a first step, it is planned to fix the still open communication issues and the issue with the generated map, and to integrate the requirements -- d), e) and g) -- that have been neglected so far. In a second step, a manufacturer of rollators is to be sought so that a prototype ready for series production can be developed in a separate application-oriented research project. In both steps, tests in the laboratory and in real environments are to be carried out.

\bibliographystyle{IEEEtran}

\section{Introduction}
 The Agebots project "Implementation of service robotics in the field of elderly care", is an interdisciplinary, federally funded project following a human-centered and iterative approach to support the elderly care with service robotic solutions. It focuses on the needs and challenges of the caregivers and the elderly in these care facilities. Different design thinking methods are used to elicit those needs and challenges, including multiple interviews with potential end-users, observations, shadowings, personae, rapid prototyping, laboratory, and field testing. Considering the different dimensions, i.e., the human, technology, business, and law aspects, prevents a one-sided, technical driven approach and promotes the implementation of requirement-based solutions.
The increase in the elderly population in Switzerland will reach an expected share of over 27\% of the 65+ population group (2.8 million) by 2050. This will cause major challenges, especially for retirement and nursing homes \cite{fur2017bevolkerung, suisse2015zukunft}. This ageing of society, in combination with work overload, changing care needs, lack of time, human errors, and economic pressure lead to a decline in the quality of care. To counteract this decline and ensure the satisfaction of the elderly and caregivers, the care facilities are dependent on innovative and financially feasible solutions \cite{klein2018robotik}. These innovative solutions should ensure a high-quality, and efficient care for the elderly in the future. Furthermore, the solutions should relieve the caregivers to give them more time for interpersonal activities. Considering these developments, service robotics might be the solution.

\section{The Agebots Project}
The Agebots project is structured in such a way that the robotic solutions are tailored to the challenges and needs of the end-users and are developed and tested in two iterations (cf. Table \ref{table_1}). The focus of the first testing was the technical feasibility and implementation. Here the robotic solutions were tested first with non-users. In the second Lab Test the robot systems were tested with end-users, i.e., elderly and caregivers of the retirement and nursing home. The focus of the second testing was on acceptance in terms of user-friendliness, readiness to use and personal impression. For each of the Lab Tests, a concept was created with specific observation criteria. The application fields could be tested several times by different users resulting in a variety of feedbacks and inputs. In addition, observations were carried out which allowed for identification of interaction difficulties with the robots. The feedback was implemented onsite (laboratory), allowing to continue the testing with the improved functionalities. Specifically, this iterative approach helped to implement and test the end-users' requirements directly, so that their wishes could be considered in the best possible way. The range of applications, which were tested in the two Lab Tests, were reduced to the most feasible or most promising ones throughout the testing phases.
\begin{table}[h]
\caption{Project Phases and general Results}
\label{table_1}
\begin{center}
\begin{tabular}{|p{2cm}|p{6cm}|}
\hline
Project phases & Results\\
\hline
\hline
Understanding & One hundred forty-four challenges for the caregivers and the elderly were identified.\\
\hline
Ideation & Fifteen application fields with 170 ideas for caregivers and ten application fields with 88 ideas for elderly were generated.\\
\hline
Technology & Seven robotic systems were procured, and functional tests were conducted.\\
\hline
Lab test 1 & Feasibility tests carried out with non-users (students, employees from the retirement and nursing home, and the universities), and first legal analysis was conducted. 25 applications were reduced to 12.\\
\hline
Iteration 1 & Application development of the robot systems based on findings of lab test 1. Research framework for the evaluation of the qualitative and quantitative data using a relational database was created and filled with insights.\\
\hline
Lab test 2 & Usability and acceptance tests with the caregivers and elderly of the specific retirement and nursing home in a laboratory setting. A business plan for the development of new markets for the implementation partner (here robot manufacturer) was elaborated.\\
\hline
Field testing (planned) & Testing of legally compliant, desirable, feasible, and economical application in real-world environments in the retirement and nursing home.\\
\hline
Marketing (planned) & "Sale" and distribution of the service robot systems in German-speaking countries by the main implementation partner (robot manufacturer).\\
\hline
\end{tabular}
\end{center}
\end{table}

\section{Companion Challenge}
One of the 144 identified challenges was the Companion Challenge which will now be discussed. Caregivers need to escort the elderlies several times a day to different locations within the retirement and nursing homes. A necessary but very time consuming activity for the caregivers, identified in the literature as the most requested feature to maintain the independence of the elderly \cite{hebesberger2015staff, pineau2003towards}. Therefore, the goal of this project is to find innovative assistance systems and robotic solutions, which can relieve or support the caregivers in this task, and preserve the elderlies independence.

\section{The Autonomous Rollator Design Process}
In this section we will discuss three iterations of the development of our autonomous rollator as an escorter robot.
\subsection{First iteration}
In the case of the Companion Challenge, one result of a creativity workshop was a rollator combined with a wheelchair. This first low-fidelity prototype was made of cardboard, cord, post-its and drawings (cf. Figure \ref{fig:experiment_1}).
At this workshop, the technical requirements for a possible solution were discussed and defined with the end-users, which were then procured, provided, and put into operation.

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{RollatorV02.jpg}

  \caption{Low-fidelity prototype of the rollator}
 \label{fig:experiment_1}
\end{figure}

\subsection{Second iteration}
In the first Lab Test (see Table \ref{table_1}), the robot systems and the evaluated application were tested with different non-users such as students and employees of universities as well as employees of the retirement and nursing home.

During the technical development, the following requirements for the Companion System were identified in accordance with the findings of \cite{hawes2017strands,pineau2003towards}:
\begin{itemize}
\item[a)]	Determining the position and orientation in space
\item[b)]	Generating a map, annotating important locations, and restricting areas
\item[c)]	Path planning considering temporary obstacles and physical variability
\item[d)]	Ensuring long-term autonomy of a system
\item[e)]	Determining the current floor
\item[f)]	Communicating with, monitoring of, and adapting to the user
\item[g)]	Ease of operation
\end{itemize}
Based on these technical requirements we adjusted the rollator from ello to escort the elderly autonomously.
In the following, we will discuss two iterations of modification of this rollator to become an autonomous robot.

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{RollatorV1.png}

  \caption{(a) Picture of the prototype in the first iteration. (b) Graphical user interface example used in the first iteration.}
 \label{fig:experiment_2}
\end{figure}
As a second iteration, we used the Wizard of Oz (WOZ) approach \cite{dahlback1993wizard}. The aim was to get an initial feeling for the requirement f) and the associated needs of the system. For this purpose, a tablet was mounted on a rollator. By means of the tablet, the wizard interacted visually and acoustically with the elderly (cf. Figure \ref{fig:experiment_2}).

The results achieved with the second prototype proved that communication via tablet works in principle. The WOZ approach allowed the wizard to interact with and actively influence the behavior of the elderly. Though, implementing this kind of interaction and influence on a computer and realizing an autonomous accompanying system with it is challenging. Additionally, it is difficult for the elderly to concentrate on the path and read the information on the tablet at the same time. This is especially problematic for people who are no longer sufficiently mobile. In summary, a communication tool other than the tablet still needs to be found.

\subsection{Third iteration}
For the second Lab Test (cf. Table \ref{table_1}), the rollator was rebuilt, as can be seen in Figure \ref{fig:experiment_3}, to tackle the previously mentioned requirements a) to c). The aim was to shift from the WOZ approach to an autonomous system and to become better acquainted with the possible applications. The rollator was equipped with two RLS RMB14 encoders connected to a BeagleBone blue single-board computer and a RPLIDAR A1 light detection and ranging sensor connected to a laptop. The BeagleBone blue and the laptop communicate over TCP/IP. Using odometry the position and orientation of the rollator is estimated. These calculations are done in the EEROS framework, and the results are sent to the ROS2 software stack \cite{thomas2014next}. To perform simultaneous localization and mapping (SLAM) the slam toolbox \cite{macenski2019use,macenski2021slam}, and for the navigation the Nav2 project \cite{macenski2020marathon} is used.
\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{RollatorV2.png}

  \caption{(a) Picture of the prototype of the third experiment. (b) System overview of the third experiment.}
 \label{fig:experiment_3}
\end{figure}
Caregivers were able to test the third prototype in a replicated elderly residence. Their feedback was positive, and they see potential in the application. From a technical point of view, the increasing computational effort required by SLAM with the size of the mapped area is a limiting factor. Another problem was that the rollator localization sometimes suddenly switched its location to the other side of a wall when it came too close to the wall. However, this was only observed in the replicated retirement and nursing home and may be understood as the walls are made of thin wooden panels. Nevertheless, this caused the map to become unusable and therefore the application had to be restarted. A possible solution would be to switch to a static map and only use SLAM for the initial recording of the map. Overall, further research is required to improve the prototype and tackle the remaining requirements.
\section{Discussion and Conclusion}
The human-centered approach in the analysis and implementation phase of technical assistance systems offers the possibility to align solutions directly with the needs of end-users. This allows corrections in the design process before the robotic solutions are implemented in the real environment. Through repeated iterative testing, prototypes are continuously adapted and improved.

Well-defined requirements for a companion system were identified and three prototypes were successfully developed to gain experience in dealing with these challenges. Problems in communicating with the elderly via a display were identified using the WOZ approach. The technical feasibility of localisation and navigation with a walker was demonstrated and the solution was evaluated by caregivers.

The results obtained in the second iteration showed that further research is needed in the identified challenges f) communicating with, monitoring of, and adapting to the user, as well as g) ease of operation. One possible solution, which will be investigated in a follow up study, is the implementation of force guided navigation. Like a lane departure warning system, the aim is to subtly point the user in the right direction. Ultimately leading to an intuitive communication that leaves the user's visual senses free for other tasks.

Although some software improvements are still needed to ensure the long-term autonomy, the prototype built in the third iteration demonstrated that escorting is possible with simple and inexpensive sensors. Furthermore, the positive oral feedback from caregivers about the usefulness of such a system is an important factor in deciding that the system should be further developed in a subsequent study.

Two steps are planned for the further investigation. In a first step, it is planned to fix the still open communication issues and the issue with the generated map, and to integrate the requirements -- d), e) and g) -- that have been neglected so far. In a second step, a manufacturer of rollators is to be sought so that a prototype ready for series production can be developed in a separate application-oriented research project. In both steps, tests in the laboratory and in real environments are to be carried out.

\bibliographystyle{IEEEtran}


\section{Introduction}
\label{intro}

A significant number of native Hindi or Malayalam speakers use Latin script instead of Devanagari or other Brahmic scripts for a wide range of social media interactions such as posting tweets, updating Facebook status, commenting on YouTube videos, and writing reviews for restaurants/movies.
This behavior occurs because keyboard optimizations focus primarily on English \cite{bi2012multilingual} and languages such as Hindi or Malayalam can be very time consuming to type on small devices. If users prefer the original script, the current solution is to back-transliterate the romanized text to the original language \cite{manglishKeyyboard}. Examples of this transliteration process for Hindi and Malayalam are shown in Figure~\ref{fig:translit}. In the first row, English and Hindi are translations of each other, while Latin-transliterated (or \textit{romanized})
Hindi is phonetically identical to the Devanagari version but written using Latin characters.

In this context, we explore
state-of-the-art language models such as multilingual BERT \cite[mBERT]{devlin2018bert} and XLM-R \cite{conneau2019unsupervised} to improve their multilingual generalizability through inclusion of romanized Hindi and Malayalam, as shown in Figure~\ref{fig:intro}. Previous work \cite{pires2019multilingual} has shown that existing transformer-based representations may exhibit systematic deficiencies for certain language pairs.
This deficiency also appears in transliterated sentences, as shown in the left panels of Figure~\ref{fig:unsup} (in Section \S\ref{results}), where the t-SNE plot of 3-way parallel datasets consisting of sentences in source (English), target (Malayalam/Hindi), and their romanizations are clearly separated in their own clusters, even though they match semantically. We also provide a t-SNE plot of mBERT/XLM-R representations showing this deficiency across various transformer layers in Appendix~\ref{app:A}.
Our work aims
to provide
a general solution towards alleviating this issue, by designing a generic and extensible architecture that can be used for aligning cross-lingual sentence representations.

\begin{figure}
    \centering
    \includegraphics[scale=0.245]{intro7.png}
    \caption{Overview of our method to enhance the multilinguality of transformer models such as mBERT or XLM-R to include Latin-transliterations (romanizations) for two Indic languages: Hindi and Malayalam.} \label{fig:intro}
\vspace{-1em}
\end{figure}

\begin{figure*} [t]
    \centering
    \includegraphics[scale=0.34]{translit8.png}
    \caption{Examples of English sentences and corresponding translations and transliterations. } \label{fig:translit}
\vspace{-1em}
\end{figure*}

Our problem setting is related to language alignment works where static \cite{smith2017offline,conneau2017word} or contextual \cite{aldarmaki2019context,schuster2019cross,cao2020multilingual} word representations from different languages are aligned to a shared vector space. Such methods primarily use a parallel word corpus and either design an alignment loss or explicitly perform rotations (transformations) on the representations. Our work deviates from these methods in three aspects: (a) we focus on sentence-level representation (in this case as produced by the \texttt{[CLS]} classification BERT token; (b) we create a \textit{synthetic 3-way} parallel corpus out of the source data using machine translations and transliterations, and (c) by using a Teacher-Student training scheme, the final representations of the 3 language variants (source, target, and romanized target) are aligned to the same source language space as produced by the original pre-trained model.

Our Teacher-Student model is inspired from knowledge distillation methods \cite{hinton2015distilling} that are intended to transfer knowledge from a complex model (Teacher) to a simpler model (Student). This approach has been utilized for various tasks, e.g. for reducing the dimension of word embeddings \cite{shin2019pupil}, distilling BERT for text generation \cite{chen2019distilling}, self-knowledge distillation \cite{hahn2019self}, or contrastive learning \cite{chen2020simple,fang2020cert}. We take a similar approach where the Teacher model acts as an anchor by freezing all its layers, while the Student model is fine-tuned based on our optimization procedure to align sentences in English with their target translations and transliterations. Such an approach is necessary, as opposed to other methods like pre-training or self-supervised learning which are outside the scope of this work, because large unlabeled transliterated datasets are not available and collecting them is non-trivial. Thus, we focus on \emph{fine-tuning} and aligning representations of already pre-trained models.

The practical implications of our work are demonstrated by applying it on naturally-occurring transliterated  datasets of tweets posted during the North India and Kerala flood crises. A model that can immediately handle transliterated tweets by producing embeddings in the same space as that of English tweets can immensely benefit information systems for emergency services, by
utilizing the vast amount of crisis response models trained in English~\cite{lewis2011crisis,united2020global,nguyen2017robust,krishnan2020unsupervised}.

\iffalse
\begin{figure*}
    \centering
    \includegraphics[scale=0.24]{tsne3.png}
    \caption{t-SNE plot of embeddings across the multi-head attention layers shows that the alignment deficiency identified in previous works \cite{pires2019multilingual} also extends to transliteration. \textit{Note}: XLM-R on Hindi (top), mBERT on Malayalam (bottom), English (blue), Original Script (green), and Latin-Transliteration (red).}
    \label{fig:tsne}
\end{figure*}
\fi

\paragraph{Contributions:} \textbf{a)} We propose a novel Teacher-Student method to address the alignment problem for contextual representations of transliterated text produced by multilingual language models (and show its efficacy on Hindi and Malayalam). \textbf{b)} We release two newly labeled datasets; a binary \textit{sentiment} dataset of Malayalam movie reviews and a binary \textit{relevancy} dataset of tweets posted during North India and Kerala flood crises.

\section{Methodology}

We first describe the problem of cross-lingual transfer for transliterations, followed by our Teacher-Student model.

\subsection{Problem Definition}

Given a source ($S$) dataset in language $\sigma$, e.g. in English (en), the goal is to train a classifier such that it can be used to predict examples from a target ($T$) dataset that consists of transliterations of the target language ($\tau$). To tackle the lack of training data in the transliterated target, as well as the lack of representation alignment between the source sentences $X^\sigma$ and the transliterated target space $X^\tau$, we propose data augmentation.
Specifically, we first create translations $X^{\tr(\sigma,\tau)}$ of the source language sentences in the target language. Then, we also create transliterations of those translated sentences into the source language's script: $X^{\tl(\tr(\sigma,\tau),\sigma)}$.\footnote{$\tr(a,b)$ represents translation from language $a$ to $b$, and $\tl(a,b)$ represents transliteration from language $a$ to $b$.} All of these are matched with the correct labels $y^\sigma$ from the original dataset.
Briefly outlined:
\begin{itemize}[noitemsep,leftmargin=*]
    \item[] \textbf{Input:}  $\mathcal{S} = X^{\sigma}$, $y^{\sigma}$
    \item[] \textbf{Augmented Input:}\\   $\mathcal{S'} = X^{\sigma}$, $X^{\tr(\sigma,\tau)}$, $X^{\tl(\tr(\sigma,\tau), \sigma)}$, $y^{\sigma}$
    \item[] \textbf{Goal:} $\mathcal{T} = y^{\tau} \gets \predict(X^{\tau})$.
\end{itemize}

For example, $X^{\tl(\tr(en,hi), en)}$ represents the data that are the result of first translating from English to Hindi, then romanizing the result. The augmentation process can be performed using any existing machine translation and transliteration tool. An example of this process is shown in Figure~\ref{fig:translit}. In the first row, second column is the result of $\tr(\text{en,hi})$ and the third column is the result of $\tl(\tr(\text{en,hi}),\text{en})$.

\subsection{Teacher-Student Model \& Joint Training}

\begin{figure}
    \centering
    \includegraphics[scale=0.28]{model4.png}
    \caption{Teacher-Student Model \& Joint Training.} \label{fig:model}
\end{figure}

\begin{figure*} [t]
    \centering

    \includegraphics[scale=0.33]{new_data_stat8.png}
    \caption{Data samples from new datasets. Malayalam and Transliterated-Malayalam movie sentiment data samples (Top). North India and Kerala floods data samples (Bottom).} \label{fig:ml_stat}
\end{figure*}

The base component of our proposed model is straightforward: it obtains sentence representations from a pre-trained language model (mBERT, XLM-R, or similar) and uses the \texttt{[CLS]} token to classify the utterance. Our Teacher-Student method uses two such models, jointly trained, as outlined in Figure~\ref{fig:model}.
In this setup, the Teacher model acts as an anchor that does not change, i.e. all its multi-head attention layers are frozen. The goal is that the representations produced from training the Student model for the translated and transliterated data will eventually align with the original (Teacher) model's source language pre-trained representations.

Training consists of two tasks, an unsupervised alignment task and a classification task. The goal of the alignment task is to ensure that the three variants (source, target, and transliterated target) end up with similar representations. As this only requires a 3-way parallel corpus with no labels, it is trained in an unsupervised fashion. The goal of the classification task is to train the model for the given class labels. Joint training on both tasks is necessary for tackling our problem

Since the two tasks we are interested in only require the classification token for making a prediction, we build our loss functions on top of the \texttt{[CLS]} token. We treat its representation as the encoder's output ($h$), which is directly used to compute the unsupervised alignment loss and passed through linear layers to compute the classification output ($p = \softmax(W h + b)$).
As shown in Figure~\ref{fig:model}, $h_{\sigma}$, $h_{\tr}$, and $h_{\tl}$ denote the representations of $X^{\sigma}$, $X^{\tr(\sigma,\tau)}$, and  $X^{\tl(\tr(\sigma,\tau), \sigma)}$ respectively.

The unsupervised alignment loss consists of three components: $\mathcal{L}_{ts}$, $\mathcal{L}_{tr}$, and $\mathcal{L}_{tl}$.  The \textit{Teacher-Student Loss} ($\mathcal{L}_{ts}$) is defined as the difference between output embeddings produced using source language ($X^{\sigma}$) by the Student ($s$) versus the Teacher ($t$).
The \textit{Translation Loss}  ($\mathcal{L}_{tr}$) is defined as the difference between output embeddings produced using $X^{\tr(\sigma,\tau)}$ on the Student versus $X^{\sigma}$ on the Teacher. Similarly, the \textit{Transliteration Loss} ($\mathcal{L}_{tl}$) is defined as the difference between output embeddings produced using $X^{\tl(\tr(\sigma,\tau), \sigma)}$ on the Student versus $X^{\sigma}$ on the Teacher.

All distances are measured using the cosine similarity between the two vectors.\footnote{We use $d(a,b) = 1 - \frac{a\cdot b}{|a||b|}$.} Essentially, all three losses penalize moving away from the Teacher's representation of the source language:
\begin{equation}
\small
\vspace{-.5em}
\begin{aligned}
\mathcal{L}_{ts}= \frac{1}{N} \sum &d(h_{\sigma}^{t}, h_{\sigma}^{s}), \hspace{.2cm} \mathcal{L}_{tr} = \frac{1}{N} \sum d(h_{\sigma}^{t}, h_{tr}^{s}) \\
&\mathcal{L}_{tl} = \frac{1}{N} \sum d(h_{\sigma}^{t}, h_{tl}^{s}), \\
\end{aligned}
\end{equation}
\noindent where $N$ represents the number of samples. Thus, the final unsupervised alignment loss ($\mathcal{L}_{u}$) defined over the Teacher-Student model can be defined as the weighted sum of the three losses:
\begin{equation}
\small
\mathcal{L}_{u} = \beta_{1} \mathcal{L}_{ts} + \beta_{2} \mathcal{L}_{tr} + \beta_{3} \mathcal{L}_{tl} \label{eq:4}
\end{equation}
Meanwhile, the loss function ($\mathcal{J}_{joint}$) for the sentiment task is a sum of binary cross-entropy (BCE\footnote{$BCE = - \frac{1}{N} \sum_{i=1}^{N} y_i \log \hat{y_i} +  (1-y_i) \log (1-\hat{y_i})$}) losses, defined similarly for the three variants of parallel data (Source ($\sigma$), Translated ($tr$), and Transliterated ($tl$)):
\begin{equation}
\small
\mathcal{J}_{joint} = \sum_{k \in [\sigma, tr, tl]} BCE_{k}\label{eq:5}
\vspace{-.3em}
\end{equation}
The overall loss is simply the combination of the unsupervised and the classification loss.
\begin{equation}
\small
\mathcal{L}_{joint\_ts} = \mathcal{J}_{joint} + \alpha \mathcal{L}_{u} \label{eq:6}
\vspace{-.3em}
\end{equation}
where $\alpha$ is the hyperparameter that controls the unsupervised alignment loss.
To summarize, the Teacher-Student model takes the augmented data ($\mathcal{S'}$) as the input and optimizes over a joint loss function that comprises of an unsupervised component that aligns the translated and transliterated representation into the same vector space as the source language and a supervised component that learns to classify for the task at hand.

\begin{table*} [ht]
\small
\centering
\begin{tabular}{lccccc}
\toprule
\multirow{2}{*}{\textbf{Dataset}} & \multirow{2}{*}{\textbf{$+$}} & \multirow{2}{*}{\textbf{$-$}} & \textbf{Avg. \# of words}  & \textbf{Avg. \# of chars.}  & \textbf{Avg. \# of chars.} \\
&  & & \textbf{per sentence}  & \textbf{per word}  & \textbf{per sentence} \\
\midrule
\multicolumn{6}{c}{\textbf{Movie Review Datasets - Sentiment}}\\
Hindi Movie Reviews  & 335 & 293 & 154.5 & 4.0 & 613.8  \\
Romanized Hindi ($\hi_{\ro})$ & 335 & 293  & 154.5 & 5.0 & 775.8  \\
Malayalam Movie Reviews & 501 & 451 & 10.4 & 9.3 & 108.2  \\
Romanized Malayalam ($\ml_{\ro})$ & 501 & 451  & 10.4 & 10.8 & 122.6  \\
\midrule
\multicolumn{6}{c}{\textbf{Crisis Tweet Datasets - Relevancy}} \\
North India Floods ($\hi_{nf}$) & 206 & 250 & 20.2 & 3.9  & 78.1   \\
Kerala Floods ($\ml_{kf}$) & 109 & 132 & 19.1  & 5.4 & 103.6   \\
\bottomrule
\end{tabular}
 \caption{Statistics for the test datasets.} \label{table:stat}
 \vspace{-1em}
\end{table*}

\section{Datasets}
In this section, we outline the datasets we use in our experiments.

\subsection{English Movie Reviews}

English movie reviews are sampled from the large IMDb movie review dataset \cite{maas2011learning} with randomly sampled balanced counts of $5000$ positive and $5000$ negative reviews for training and $500$ each for validation. During training, this dataset is translated to Hindi and Malayalam, and subsequently transliterated. Ground truth Hindi and Malayalam datasets (described in the following sections), are used only for evaluation. The `$\ro$' notation used in the following section for representing datasets signify $\tl(\bullet ,\text{en})$; i.e. romanization or Latin-transliteration using transliteration tools.

\subsection{Hindi Movie Reviews ($\hi$, $\hi_{\ro}$)}
\label{hi_data}

The original Hindi movie review dataset\footnote{\url{https://www.kaggle.com/disisbig/hindi-movie-reviews-dataset}} is constructed from various News Websites. This dataset consists of positive, negative, and neutral movie reviews. We select only positive and negative reviews from both training and validation datasets to construct the test dataset for our cross-lingual setup. From this, we construct the romanized Hindi dataset using the Indic-nlp transliteration tool~\cite{kunchukuttan2020indicnlp}.\footnote{\url{https://pypi.org/project/indic-transliteration/}}

\subsection{Malayalam Movie Reviews ($\ml$, $\ml_{\ro}$) - New Dataset I}
\label{ml_data}

For Malayalam movie reviews, we construct a new human-labeled dataset from the Samayam News website.\footnote{\url{https://malayalam.samayam.com/malayalam-cinema/movie-review/articlelist/48225004.cms}} Reviews are lengthy, in general, with plenty of neutral text. So, a native Malayalam speaker was tasked with extracting a few sentences from each movie review such that each positive or negative example in our dataset is highly polar. From this, we construct the romanized Malayalam dataset using the \texttt{ml2en} tool.\footnote{\url{https://pypi.org/project/ml2en/}} A few samples from the dataset are shown in Figure~\ref{fig:ml_stat} and dataset statistics are available in Table~\ref{table:stat}.

\subsection{Crisis Tweets ($\en$)}
Appen\footnote{\url{https://appen.com/datasets/combined-disaster-response-data/}} provides a labeled collection of tweets posted during various natural disasters such as earthquakes, floods, and hurricanes. The three most common languages in this dataset are English, Spanish, and Haitian Creole. For our experiments, we focus on the \texttt{related} label column signifying the relevancy of tweets. The dataset also contains English translations of non-English tweets. Training and validation datasets are constructed using only the English tweets (\texttt{message} column in the Appen dataset). Statistics are shown in Table \ref{table:stat}.

\subsection{Crisis Tweets: North India ($\hi_{nf}$) and Kerala ($\ml_{kf}$) Floods - New Dataset II}

To construct our test datasets for the crisis tweet classification experiments, we collected tweets from the 2013 North India and 2018 Kerala Floods using Twitter API.  We filtered these tweets to restrict only to naturally-occurring transliterated Hindi and Malayalam sentences, using a set of transliterated crisis-related keywords such as \textit{madad}, \textit{toofan}, \textit{baarish}, \textit{sahayta}, \textit{floods}, etc., for Hindi and \textit{pralayam}, \textit{vellapokkam}, \textit{vellam}, \textit{sahayam}, \textit{durantham}, \textit{veedukal}, etc., for Malayalam. With the help of a native language expert for each language who are also proficient in English, the tweets are labeled based on contextual-relevancy or relatedness; similar to the \texttt{related} label for the English tweets in the Appen dataset. Dataset statistics are shown in Table~\ref{table:stat} and a few tweet samples are shown in  Figure~\ref{fig:ml_stat}.

\section{Experimental Setup}
Since we are interested in cross-lingual transfer, we only use the English datasets (IMDb reviews for Sentiment Analysis and Appen Crisis Dataset for Tweet Classification) for training, augmented with their automatic translations and transliterations.
We evaluate on all other datasets (c.f. Table~\ref{table:stat}).

\paragraph{Monolingual LM Baselines.}
Our first baseline uses pre-trained language models from Hugging Face \cite{wolf2020transformers} that are monolingually trained on Hindi\footnote{\url{https://huggingface.co/monsoon-nlp/hindi-tpu-electra}} and Malayalam.\footnote{\url{https://huggingface.co/eliasedwin7/MalayalamBERT}}

\paragraph{mBERT/XLM-R Baselines.}
We also consider baselines using multilingual masked LMs (MLM), specifically mBERT \cite{devlin2018bert} and XLM-R \cite{conneau2019unsupervised}.
We compare our models with the following MLM baselines: \textbf{1)} a model trained using only in English, \textbf{2)} a model trained using English translated to the target language using MarianMT \cite{mariannmt}, \textbf{3)} a transliterated model which is trained using the target-transliterated
version of target-translated English data, and \textbf{4)} a combination of the three. These baselines use our augmented datasets but do not use the Teacher-Student training scheme, e.g. $\text{mBERT}_{tl}$ refers to mBERT trained using target-transliterations of target-translated English data.

\begin{table*}[t]
\small
\begin{center}
\begin{tabular}{c*{2}{c}*{2}{c}*{2}{c}*{2}{c}|*{2}{c}}
\toprule

   \textbf{Test Data} $\rightarrow$  & \multicolumn{2}{c}{$\mathbf{hi_{ro}}$} & \multicolumn{2}{c}{$\mathbf{ml_{ro}}$}  & \multicolumn{2}{c}{$\mathbf{hi_{nf}}$} & \multicolumn{2}{c}{$\mathbf{ml_{kf}}$} & \multicolumn{2}{c}{AVG}\\

\textbf{Models}  $\downarrow$ & Acc. & F1 & Acc. & F1 & Acc. & F1 & Acc. & F1 & Acc. & F1 \\
\midrule
Monolingual LM$^\diamondsuit$ & 50.38 & 37.99 & 48.76 & 47.55 & 54.39 & 49.06 & 63.35 & 63.33 & 54.22 & 49.48  \\
\multicolumn{9}{c}{\textbf{mBERT Baselines}}  \\
mBERT$_{en}$  & 47.55 & 41.23  & 48.23 & 41.63 & 56.67 & 55.27 & 57.84 & 57.72 & 52.57 & 48.96 \\
mBERT$_{tr}$  & 51.81 & 48.21  & 51.32 & 45.62 & 52.89 & 47.89  & 58.26 & 57.54 & 53.57 & 49.82 \\
mBERT$_{tl}$  & 55.29 & 54.18  & 61.72 & 56.47 & 56.14 & 55.78 & 58.09 & 57.79 & 57.81 & 56.06 \\
mBERT$_{en+tr+tl}$  & 55.16 & 54.75  & 61.72 & 61.25 & 56.71 & 56.03 & \textbf{63.74} & \textbf{63.42} & 59.33 & 58.86 \\
\multicolumn{9}{c}{\textbf{Our mBERT models}}  \\
mBERT-Joint  & 55.57 & 55.56  & 64.29 & 63.58  & 57.75 & 56.73 & 51.85 & 53.98 & 57.37 & 57.46 \\
mBERT-Joint-TS  & \textbf{57.37}$^\spadesuit$ & \textbf{57.36}$^\spadesuit$  & \textbf{65.15}$^\spadesuit$ & \textbf{65.78}$^\spadesuit$ & \textbf{63.22}$^\spadesuit$ & \textbf{63.14}$^\spadesuit$ & 62.55 & 62.40 & \textbf{62.07} & \textbf{62.17} \\
\midrule
\multicolumn{9}{c}{\textbf{XLM-R Baselines}} \\
XLM-R$_{en}$   & 50.57 & 45.76  & 50.86 & 47.13 & 58.11 & 56.77  & 61.74 & 60.98 & 55.32 & 52.66 \\
XLM-R$_{tr}$  & 49.52 & 47.67  & 51.72 & 50.16 & 57.11 & 56.95 & 61.74 & 61.72 & 55.02 & 54.13\\
XLM-R$_{tl}$  & 54.81 & 53.72  & 51.67  & 54.71 & 51.45 & 51.24  & 59.84 & 59.23 & 54.44 & 54.73\\
XLM-R$_{en+tr+tl}$ & 55.57 & 54.19 & 62.46 & 61.52  &  56.40 & 55.67 & 63.24 & 63.10 & 59.42 & 58.62\\
\multicolumn{9}{c}{\textbf{Our XLM-R Models}} \\
XLM-R-Joint  & 56.09 & 55.40 & 62.90 & 63.14 & 53.68 & 52.81  & 62.79 & 62.77 & 58.87 & 58.53\\
XLM-R-Joint-TS & \textbf{57.70}$^\spadesuit$ & \textbf{57.03}$^\spadesuit$ & \textbf{65.93}$^\spadesuit$ & \textbf{65.71}$^\spadesuit$ & \textbf{58.39} & \textbf{57.86} & \textbf{64.87}$^\spadesuit$ & \textbf{64.87}$^\spadesuit$ & \textbf{61.72} & \textbf{61.37}\\
\bottomrule
\end{tabular}
\end{center}
 \caption{Performance evaluation (weighted F1) on various romanized datasets shows that our Teacher-Student model outperforms the baselines.  $\diamondsuit$: Dedicated Hindi and Malayalam language models (LM) from Hugging Face; results reflect the best performing model by varying the training data. $^\spadesuit$: The difference is significant with p $<$ 0.05 using Tukey HSD (compared against the best baseline model).} \label{table:scores}
\end{table*}

\begin{table}[t]
\small
\centering
\begin{tabular}{cc|cc}
\toprule
   \textbf{Train Language} $\rightarrow$ & $\mathbf{en}$ & $\mathbf{hi}$ & $\mathbf{ml}^\dagger$      \\
\textbf{Model} & (Baseline) & \multicolumn{2}{c}{(Joint-TS Model)}\\
\midrule
mBERT & 81.06 & 82.35  & 71.25 \\
XLM-R & 83.47 & 84.00  & 74.16    \\
\bottomrule
\end{tabular}
 \caption{Evaluation (weighted F1) on IMDb English test data showing that our model preserves the performance on the source language for Hindi but not for Malayalam. The results in $\hi$ and $\ml$ are with our modifications. $\dagger$: See discussion.} \label{table:scores2}
\end{table}

\iffalse
\begin{table}[t]
\small
\begin{center}\begin{tabular}{cc}
\toprule
   \textbf{Test Language} $\rightarrow$ & $\mathbf{en}$      \\
\midrule
mBERT$_{en}$ & 81.06    \\
XLM-R$_{en}$ & 83.47    \\
\end{tabular}
\begin{tabular}{ccc}
\midrule
   \textbf{Test Language} $\rightarrow$ & $\mathbf{en_{[k=hi]}}$   & $\mathbf{en_{[k=ml]}} ^\dagger$       \\
\midrule
mBERT-Joint-TS$_{k}$  & 82.35  & 71.25 \\
XLM-R-Joint-TS$_{k}$  & 84.00  & 74.16    \\
\bottomrule
\end{tabular}
\end{center}
 \caption{Evaluation (weighted F1) on IMDb English test data showing that our model preserves the performance on the source language for Hindi but not for Malayalam. $\dagger$: See discussion.} \label{table:scores2}
\end{table}
\fi

\begin{table}[t]
\small
\begin{center}
\begin{tabular}{ccc}
\toprule
   \textbf{Test Language} $\rightarrow$   & $\mathbf{hi}$ & $\mathbf{ml}$     \\
\midrule
mBERT$_{\en}$ & 54.34  &  54.07  \\
mBERT$_{\tr}$ & 60.11 &  66.20  \\
mBERT-Joint-TS  & 61.35  & 66.24 \\
\midrule
XLM-R$_{\en}$ &  60.82 & 78.11 \\
XLM-R$_{\tr}$ &  59.72 & 71.40 \\
XLM-R-Joint-TS & 62.23 & 76.46$^\dagger$\\
\bottomrule
\end{tabular}
\end{center}
 \caption{Evaluation (weighted F1) on Hindi and Malayalam original script showing that our model preserves or outperforms their performance on the baselines except for Malayalam on XLM-R. $\dagger$: See discussion.} \label{table:scores3}
\end{table}

\iffalse
\begin{table}[t]
\small
\begin{center}
\begin{tabular}{ccccc}
\hline
   \textbf{Test Data} $\rightarrow$  & $nf$ & $kf$ & $hi_{fb}$ & $ml_{yt}$       \\
\hline
LM Baseline &  &   &  &  \\
\multicolumn{5}{c}{\textit{mBERT Models}}  \\
mBERT$_{en}$ &  55.27   & 57.72 & &   \\
mBERT$_{tr}$ &  47.89   & 57.54 & & \\
mBERT$_{tl}$ & &  &  &  \\
mBERT$_{en+tr+tl}$ &  &   &  &  \\
mBERT-Joint &  &   &  &  \\
mBERT-Joint-TS &  &   &  &  \\
\hline
\multicolumn{5}{c}{\textit{XLM-R Models}}  \\
XLM-R$_{en}$ &  56.77  & 60.98 & &\\
XLM-R$_{tr}$ &  56.95 & 61.72 & &\\
XLM-R$_{tl}$ &  51.24  & 59.23 & &\\
XLM-R$_{en+tr+tl}$ &  55.67  & 63.10 & &\\
XLM-R-Joint &  52.81 & 62.77 & &\\
XLM-R-Joint-TS & \textbf{57.86} & \textbf{64.87} & &\\
\hline
\end{tabular}
\end{center}
 \caption{Performance evaluation (weighted F1) on Social Media Datasets shows that our model outperforms the baselines.} \label{table:scores_crisis}
\end{table}
\fi

\paragraph{Our Proposed Models.}

\textbf{Joint-TS} represents our Teacher-Student model shown in Figure \ref{fig:model} and Eq.~\ref{eq:6}. We also perform an ablation (the \textbf{Joint} model) without the Teacher model (setting $\alpha=0$ in Eq.~\ref{eq:6}), i.e. this model does not have an anchored Teacher, which means that there is no penalty for representations of parallel sentences being different.

\paragraph{Implementation Details.}

Our implementation is in PyTorch  \cite{NEURIPS2019_9015} with the transformers library~\cite{wolf2020transformers}. We use the pre-trained cased multilingual BERT and the pre-trained \textit{xlm-roberta-base} with a standard sequence classification architecture.
Maximum epoch is set to~$40$ with an early stopping patience of~$10$, batch size to~$32$, and we use the Adam optimizer \cite{kingma2014adam} with a learning rate of~$5\mathrm{e}{-5}$. We select the best model based on the validation set, using both accuracy and weighted F1 as performance measures.

\section{Results \& Discussion}
\label{results}

\paragraph{Performance on Transliterated Target.} Table \ref{table:scores} shows the classification performance\footnote{Model runtime shown in Appendix \ref{app:B}.} on transliterated datasets. Averaging the scores, we see a total boost of $+5.6\%$ on mBERT and $+4.7\%$ on XLM-R in weighted F1 performance across the 4 romanized datasets.\footnote{With similar trends on accuracy as well.}
Our top baseline models in each of the masked language models are mBERT$_{en+tr+tl}$ and XLM-R$_{en+tr+tl}$, which combine the augmented data with the original data to fine-tune the model for the classification task. The improvement produced by our model is due to the fact that the pre-trained models have likely not seen that many transliterations in these languages. mBERT is trained using data from Wikipedia, while XLM-R uses Common Crawl. As such, XLM-R likely has been trained on at least some code-switched or transliterated data. This is also supported by the scores in Tables~\ref{table:scores}, \ref{table:scores2}, and \ref{table:scores3}, where XLM-R$_{en}$ outperforms mBERT$_{en}$ on all datasets, i.e. XLM-R's English representations are much more generalizable than mBERT's for these two languages.

\begin{figure}[t]
    \centering
    \includegraphics[scale=0.18]{align_comp2.png}
    \caption{The visualization of final layer representations (from the movie review test data) shows the effectiveness of unsupervised alignment training in bringing the representations of the 3 variants in a shared space. } \label{fig:unsup}
    \vspace{-1em}
\end{figure}

\paragraph{Performance on Original-Script Target.} Beyond improving on transliterated text, we need our model to handle original, not-transliterated text equally well. Evaluating on the original script (Table~\ref{table:scores3}) we find that our models do preserve or even outperform the baselines trained only with English or non-romanized text, especially in Hindi.
The exception is Malayalam in XLM-R, which could be attributed to the power of XLM-R pre-training in producing a more generalizable English representation than the original script ($\text{XLM-R}_{\en}$ vs. $\text{XLM-R}_{tr}$).

\paragraph{Performance on English.}  Ideally, our Teacher-Student model will preserve the performance on the source language  (in this case English).  Table~\ref{table:scores2} shows the performance evaluation on English data ($1000$ randomly selected positive and negative reviews from the IMDb dataset \cite{maas2011learning}). We observe that fine-tuning on Hindi preserves and slightly improves performance in English, but Malayalam does not ($\dagger$ in Table~\ref{table:scores2}). We speculate that, for Malayalam, this is because the transliterated embeddings (red) are not blended in well with the others compared to those for Hindi as shown in Figure~\ref{fig:unsup} (right panels).

\iffalse
\paragraph{Performance on Source and Target.} Table~\ref{table:scores2} shows the performance evaluation on English data ($1000$ randomly selected positive and negative reviews from the IMDb dataset \cite{maas2011learning}). Table \ref{table:scores3} reports the performance evaluation on the original script. These evaluations let us assess  whether our Teacher-Student model preserves the performance on the source and non-romanized target languages.
We observe that fine-tuning on Hindi preserves and slightly improves performance in English, but training on Malayalam does not ($\dagger$ in Table~\ref{table:scores2}; c.f. two bottom columns.) We speculate that this is because, for Malayalam, the transliterated embeddings (red) are not blended in well with the others compared to those for Hindi as shown in Figure~\ref{fig:unsup} (right panels). On the other hand, evaluation on the original script (Table~\ref{table:scores3}) shows that our model either preserves or outperforms the baselines trained in English and non-romanized target,  except for Malayalam on XLM-R ($\dagger$ in Table~\ref{table:scores3}). This exception can be attributed to the power of XLM-R pre-training in producing a more generalizable English representation than the original script ($\text{XLM-R}_{\en}$ vs. $\text{XLM-R}_{tr}$).
\fi

\paragraph{Representation Alignment.} Figure~\ref{fig:unsup} shows a visual reason behind our model's aforementioned performance improvements. While the t-SNE projection of the (3-way parallel) sentence representations for the original model are quite distinct based on language/script (left), our model brings all representations in the same vector space (right). This method of unsupervised training ($L_{u}$ in Eq. \ref{eq:4}) is very generic in nature that it can be adapted for any alignment scenarios where different variants or dialects of the same language need to be aligned.

\paragraph{Ablation Studies.} We conduct $8$ ablations in total. First four mBERT/XLM-R models in Table~\ref{table:scores} are ablations that show how the three language variants and their combination perform. The joint model without Teacher-student in Table~\ref{table:scores} represents the version without the alignment training. Table~\ref{table:scores2} shows performance on English and Table~\ref{table:scores3} shows performance on original script. The impact of unsupervised alignment is shown in Figure~\ref{fig:unsup} showing where the boost is coming from. Ablation of the Joint-TS model without $tr$ is not shown as our goal is to construct a single model that preserves performance across the three variants.

\begin{figure} [t]
    \centering
    \includegraphics[scale=0.42]{freezing5.png}
    \caption{Freezing up to layers $8$ to $10$ of mBERT showed to be optimal for our task.} \label{fig:freeze}
    \vspace{-1em}
\end{figure}

\paragraph{Unsupervised Domain Adaptation.} In addition to being cross-lingual, our approach also falls under the paradigm of unsupervised domain adaptation. Our sources and targets do not strictly fall in the same domain, even though the classification tasks are similar. For example, the training data for classifying tweets consists of not only floods, but also other events such as hurricanes, fires, earthquakes, etc. This leads to another application of our method, which can be deployed at the onset of crisis events in any region, with minimal requirement to collect labeled data from the new crisis, or
converting the data to native script or English,
which might save precious time for crisis response.

\paragraph{Hyperparameter Tuning.} Primarily, we tune two hyperparameters: a) $\alpha$ - the weight that is given to unsupervised alignment loss in Eq. \ref{eq:6} and b) number of layers to freeze to identify the appropriate amount of pre-trained information to be preserved without alteration. $\alpha$ values are tuned using a simple grid search from a range of [$0.01$-$1.0$]. All $\beta$ values (Eq. \ref{eq:4}) are set to $1$ to prioritize the three variants (source, target, transliterated target) equally. For Joint-TS (Eq. \ref{eq:6}), best hyperparameters for mBERT are $\alpha_{\hi_{\ro}}$= $0.3$ and best $\alpha_{\ml_{\ro}}$= $0.05$ and for XLM-R are $\alpha_{\hi_{\ro}}$= $0.5$ and best $\alpha_{\ml_{\ro}}$= $0.01$. Intuitively, we find that giving the classification loss primary focus and the unsupervised alignment loss secondary focus produced better results. On the other hand, we found that freezing bottom layers
leads to better results, as shown in Figure~\ref{fig:freeze}. The optimal amount of layers to freeze was empirically between $8$ to $10$.

\section{Related Work}

Sentiment analysis in Hindi spans a variety of tasks such as analysis of movie reviews \cite{nanda2018sentiment}, building subjective lexica for product reviews and blogs \cite{arora2013sentiment}, analysis on tweets \cite{sharma2015practical}, aspect-based sentiment \cite{akhtar2016aspect}, predicting elections~\cite{sharma2016prediction}, and analysis on code-mixed text~\cite{joshi2016towards}. Code-mixing is another related phenomenon where multilingual speakers alternate between languages, often in the same sentence. Both code-mixing and transliteration are studied for Hindi and Marathi texts using supervised learning methods by \citet{ansari2018sentiment}. We restrict our analysis to transliteration, although our dataset may contain code-mixed text. Recently, \citeauthor{khudabukhsh2020harnessing} have proposed a pipeline to sample code-mixed documents using minimal supervision. In cross-lingual context, researchers have used linked WordNets~\cite{balamurali2012cross} and cross-lingual word embeddings~\cite{singh2020sentiment} using MUSE~\cite{conneau2017word} and VecMap~\cite{artetxe2018robust} to bridge the language gap, later addressing code-mixing and transliteration. With the advent of large pre-trained language models, we take a step further in this direction to enhance mBERT/XLM-R to cover transliterations for fine-tuning it to the downstream task of sentiment analysis.

Malayalam has also seen several works on sentiment analysis~\cite{nair2015sentiment,kumar2017sentiment,nair2014sentima,ashna2017lexicon}. Recently, a new Malayalam-English code-mixed corpus~\cite{chakravarthi2020sentiment} has been constructed by scraping YouTube comments. This corpus primarily consists of romanized sentences with some code-mixing. After converting this dataset into its original script to obtain the parallel corpus, this can also be used as an additional dataset for our model evaluation.

Translate and train has been a popular methodology \cite{shah2010synergy, yarowsky2001inducing, ni2017weakly, xu2020end} that utilizes the power of existing Machine Translation tools \cite{wu2016google,mariannmt} to perform cross-lingual tasks by augmenting the original source dataset with the target-translated data before training. This kind of training could enhance the performance of multilingual representations by fine-tuning the pre-trained models such as mBERT or XLM-R, creating a pseudo-supervised environment where the model now has access to data in the target language. We follow the same approach to create strong baselines as well as for the Teacher-Student model.

\section{Future Work}
We acknowledge that existing Hindi and Malayalam translation and transliteration tools pose limitations and may cause cascading errors. For example, the Indic transliteration tool adds an extra, unnecessary \textit{`a'} for some Hindi words (eg., \textit{`sandarbh'} vs \textit{`sandarbha'}) which may not reflect how native users tend to write. Despite its limitations, using it is still beneficial to construct a strong baseline. We expect that improved transliteration systems would further improve downstream accuracy. This is also the case for the many Indic languages where extensive datasets are not available. We plan to expand our model to other Indic languages such that their translations and transliterations are aligned within the language models such as mBERT/XLM-R. Another future direction is to perform an in-depth sensitivity analysis over the $\beta_{2}$ and $\beta_{3}$ parameters that tune the unsupervised alignment loss ($L_{u}$) in Eq.~\ref{eq:4}, which might also address some of the exceptions shown in Tables~\ref{table:scores2} and \ref{table:scores3}, to ensure that the model preserves performance on English or non-romanized target data.

\section{Conclusion}
We propose a Teacher-Student model to enhance the multilinguality of language models such as mBERT/XLM-R so that it can be adapted to perform cross-lingual text classification tasks for \textit{transliterated} Hindi and Malayalam. Experiments show that our model outperforms traditional fine-tuning and other baselines built on the state-of-the-art. Furthermore, we release two human-annotated datasets: a highly polar Malayalam movie review dataset for sentiment analysis and a dataset of Hindi and Malayalam romanized tweets posted during North India and Kerala floods.
Additionally, our method presents a generic and extensible architecture that could be adapted to any language alignment scenarios where large pre-trained multilingual models may fall short.\\

\section{Acknowledgement}
We thank U.S. National Science Foundation grants IIS-1815459 and IIS-1657379 for partially supporting this research. We thank Ming Sun, Alexis Conneau, Sneha Mehta, and Raj Patel for giving valuable insights on language models, machine translation, and multilingual model training. We thank Chandini Narayan and Sujay Das for data annotation. We also acknowledge ARGO team as the experiments were run on ARGO, a research computing cluster provided by the Office of Research Computing at George Mason University.