text:
\section{Introduction}
\input{introduction/introduction.tex}

\section{Related Work}
\input{related_work/related_work.tex}

\section{The WebNLG Challenge}
\input{webnlg_challenge/webnlg_challenge.tex}

\section{Proposed Approaches}
\input{approach/approach.tex}

\section{Training}
\input{training/training.tex}

\section{Experiments}
\input{experiments/experiments.tex}

\section{Conclusion}
\input{conclusion/conclusion.tex}
\section{Acknowledgment}
This work was partially funded by the ANR Cifre conventions N°2020/0400 and N°2018/1559 and Orange-Labs Research. This work was granted access to the HPC resources of IDRIS under the allocation \textit{2020-AD01101126} made by GENCI. We also thank Thibault Cordier and Imen Akermi for the fruitful discussions.

\subsection{The Transformer Model}
\input{transformer/transformer.tex}

\subsection{Data Augmentation}
\input{data/data_augmentation.tex}

\subsection{WebNLG Dataset}
\input{data/webnlg_data.tex}

\subsection{Data Augmentation}
\input{data/data_augmentation.tex}
\subsubsection{Sentence Collection}
We make use of Wikipedia that offers a massive and free amount of data to be leveraged. The WebNLG dataset was constructed from DBPedia database which itself relies on Wikipedia. The substantial advantage of including new Wikipedia content lies in that new named entities can then be encountered from much more heterogeneous domains. This allows for a better generalization and knowledge integration. As observed in language models \cite{devlin-etal-2019-bert, radford2019language, zhang-etal-2019-ernie}, knowledge is somehow assimilated by the model throughout training.

We gathered 13,614 Wikipedia pages containing about 103 million sentences totally\footnote{We used the \textit{20200401} Wikipedia dump $\sim$ 18GB}. To alleviate the training process, we filter sentences that may impede generation quality. We discard a sentence (1) if it does not start with a upper case letter and does not end with a period, (2) if its length is longer than 50 or (3) if it contains special characters. After this first selection, 57 million (unique) sentences remain. Henceforth, we coin this set of sentences WS1.

\subsubsection{Triple Extraction}
Triples extraction is challenging and non-trivial, demanding a sharp understanding of linguistic structure. For simplicity (and time-saving), we take advantage of the Stanford Open Information Extraction (Stanford OpenIE) tool\footnote{Available at \url{https://nlp.stanford.edu/software/openie.html}} to extract triples from WS1 sentences. Stanford OpenIE package is schema-free. That means no preliminary definition of the possible predicates is required, as opposed to usual RDFs extractor. The raw text linking two entities will be retrieved as the predicate. Unfortunately, the returned triples may be incomplete, false or alike. For example, for the input sentence \textit{"Barack Obama was born in Hawaii."}, the returned triples by Stanford OpenIE are  $\langle \textit{Barack Obama; was; born} \rangle$ and $\langle \textit{Barack Obama; was born in; Hawaii} \rangle$. The first output triple is spurious besides expressing somehow the same insight than the second triple. As a result, a filtering step is crucial to reduce false triples. We fashion simple rules to limit redundancy by comparing each retrieved triple with each other. Let $t_{i}$ and $t_{j}$ be two retrieved triples, with $i \ne j$. We linearize each triple by concatenating words from the subject, predicate and object. The linearized version of $t_{i}$ and $t_{j}$ are named $t_{i}^{l}$ and $t_{j}^{l}$, respectively. To detect if $t_{i}^{l}$ and $t_{j}^{l}$ are equivalent, we first verify if either $t_{i}^{l} \in t_{j}^{l}$ or $t_{j}^{l} \in t_{i}^{l}$. We further compute the \emph{Levenshtein distance} \cite{levenshtein-distance} to consider minor variations between triples. Two triples with an edit distance smaller or equal to 2 are also believed to be similar. If triples similarity is confirmed, the longest triple is kept because sharing the more information with the input sentence. The higher the lexical coverage, the better. Indeed, the risk is to miss essential information from the sentence. If input triples are considerably incomplete, the model will suffer from hallucination, \textit{i.e.} generating content not present in the given input.

However, when $t_{i}^{l}$ and $t_{j}^{l}$ are too different but semantically identical, more sophisticated conditions should be exploited. For this purpose, we make use of BLEU \cite{papineni-etal-2002-bleu}. Extracted triples are derived from the same sentence. Therefore, a \textit{n-gram} based metric such as BLEU is a good choice to check if two triples are alike. We assume triples to be analogous if their BLEU score is greater than 50. In such case, we keep the triple maximizing its BLEU score with the reference sentence for better coverage, as explained previously. The final collection of remaining triples-sentences pairs is called ST1.

Nevertheless, the detection of erroneous information conveyed by the triples remain hard to operate. We are aware that mistaken triples jeopardize directly the performance of our model. That is why we leverage these augmented data in a pre-training step, as detailed in Section \ref{pretraining_section}.

\subsection{Training and decoding settings}
In the following, we present the preprocessing, the training phase and the decoding pipeline.

\paragraph{Preprocessing}
For preprocessing, we used the Moses tokenizer\footnote{Available on github \url{https://github.com/moses-smt/mosesdecoder}} and subword segmentation following \newcite{sennrich-etal-2016-neural} with the subword-nmt
library\footnote{Available on github \url{https://github.com/rsennrich/subword-nmt}}. We left both the input triples and output text true-cased.

In the context of the WebNLG challenge, we considered a transduction strategy \cite{gammerman-transduction}. We restricted the vocabulary to the WebNLG dataset (training, validation and testing set). This learning scheme aims at performing well on a specific set, and not necessary to generalize.

The training process remained the same for all our experiments. Only the data loaded in the batches during training were changed.

To deal with the RDF triples format, we added to the vocabulary four special tokens, namely \angles{object}, \angles{subject}, \angles{predicate} and \angles{eot} (end of triple), that we used as separators within and between triples. In the case of multiple triples, we built the Transformer input sequence by concatenating triples one after the other. We used this input format on both ST1 and WebNLG.

For the denoising pre-training, we used the WS1 dataset. The transformed sentences (see Section \ref{pretraining_section}) were directly fed to the Transformer. The model thus has to reconstruct the incomplete sentences.

Examples of the preprocessed samples for the different training settings are reported in Table \ref{table:ex_data}.

\paragraph{Training}

We used the Transformer implementation from \textsc{fairseq} library \cite{ott2019fairseq}\footnote{We used the compiled version 0.9.0 from \url{https://github.com/pytorch/fairseq}} with the \emph{transformer\_base} hyper-parameters set from \cite{vaswani-etall-transformer}. We optimized the weights of our neural networks using an \textsc{Adam} optimizer and a label smoothed cross entropy loss.

We made 10 epochs of pre-training and stopped fine-tuning when the performance with the BLEU score on the validation set did not improved after 30 epochs. At the end, we kept the model that achieved the best \textsc{BLEU} on the validation set.

To study the impact of curriculum learning, we launched fine-tuning with and without it. In the former case, we prevented the shuffling of the batches for 30 epochs. The data being sorted by number of triples, the model had to deal first with simple samples, and then with more complex ones as the learning did progress. In the results, we report this setting as CL (for Curriculum Learning).

\paragraph{Decoding}
For decoding, we did a beam search with a beam of size 5. We merged the subwords back into words and detokenized.

\subsection{Evaluation setting}
For evaluation, we used the official WebNLG evaluation script\footnote{Available on github \url{https://github.com/WebNLG/GenerationEval}}. The metrics we used to compare our models are BLEU, METEOR, chrf++ and BLEURT. BLEURT metric has been recently proposed and proven to be well-correlated with human judgments. Relying on BERT's contextual embeddings, BLEURT offers semantically robust feedback. The \emph{n-gram}-based evaluation techniques such as BLEU, METEOR or chrf++ are additional metrics to judge the generation quality. When used together, they give good assessment of the generation quality of our system.

Unlike in 2017, this time the generated sentences must be detokenized and true-cased. Our models are therefore not directly comparable with the models of the previous version of the challenge.

Also, as the test base is not available, we do not know how many references our generated sentences are compared to.

\subsection{Results}
\label{results_section}

\begin{table*}[hbt!]
\centering
\input{table/results_div3}
\caption{\label{results_table}
Ablation study: Automatic results on the official WebNLG test set for different learning strategy. For each experiment, we provide testing performance on seen categories (top), unseen entities (middle) and unseen categories (bottom). Bold and underlined values correspond to the best and second-best results respectively.
}
\end{table*}

To gauge the influence of our different learning approaches, we conducted an ablation study. We defined our baseline as the Transformer solely trained on WebNLG dataset, without curriculum. We want to assess the influence of pre-training, data augmentation and curriculum learning compared to the baseline. To provide a fair and detailed analysis, we evaluated models on seen and unseen domains to shed light on the models' generalization ability. Results are given in Table \ref{results_table}.

In Table \ref{results_table}, when comparing our baseline with pre-training strategies for each category (no curriculum), we note an average rise of 3.07, 19.6 and 10.97 respectively in BLEU when pre-training. Similar variations can be noted with METEOR, chrf++ and BLEURT metrics, albeit less striking. Based on BERT's contextual embeddings, BLEURT gives a good estimate of the semantic correlation between prediction and references. Most of the time, \emph{n-gram}-based and semantic metrics show perfect harmony. Top values of BLEURT are obtained for the same models than \emph{n-gram}-based metrics. Therefore, all metrics tend to correlate proving a good agreement. The high improvements over unseen domains are easily explained due to the diversity of augmented data. New entities and domain-specific lexicon encountered better help to model out-of-distribution data relations. Thus, such results underline usefulness of external corpora and strengthen the need of pre-trained model to lexicalize KBs.

On seen categories, our baseline give an acceptable BLEU score of 55.24. However, for out-of-domain generation, all models demonstrate severe shortcomings. Tested on unseen entities, our baseline shows a BLEU drop of 42.34 to reach 12.9 BLEU. We witness similar and even more important loss in unseen categories. In the case that predicates are unknown to the model, it is hard to generate consistent description of the input RDFs. From seen categories to unseen categories, our baseline is nearly penalized by a factor of 5. Such effect is tempered with our pre-training approaches. The average drop in BLEU of pre-trained models (without curriculum learning) is 25.6 from seen categories to unseen entities, and 35.5 from seen to unseen categories.

Unexpectedly, when a curriculum learning approach is used, we witness drops in performance. This is counter-intuitive and opposed to previous experiment on our validation set. CL seems to help lightly when the model was pre-trained with both external corpora WS1 and ST1. We propose to investigate the reason of such outcome in future work.

Interestingly, best results are revealed with a pre-training on ST1, exclusively. With 5 times less data, ST1 leads to better performance. The extracted triples surely include inaccurate triples. Notwithstanding the imperfect quality of the ST1 dataset, its use contributes to generalization ability.
On the contrary, we report that denoising pre-training does not show satisfying results when combined with our pre-training on the ST1 dataset, leading eventually to negative effect. This may be caused by an input distribution discrepancy between WS1 and ST1 input. Denoising pre-training doesn't require triples as input but a noisy sentence. Mismatch between this representation and the triples linearization may be the culprit of such side-effect.

As for the WebNLG 2020 challenge, participants were requested to submit a single model for evaluation. In our case, we decided to submit our Transformer pre-trained with both WS1 and ST1 (without curriculum). After an analysis of the generated text, the model trained with much more data has a tendency to be much more fluent and aggregate information better. \citet{castro-ferreira20:_2020_bilin_bidir_webnl} report human evaluation based on different criteria: data coverage, relevance, correctness, text structure and fluency. For each criterion, a value between 0 (complete disagreement) and 100 (complete agreement) is given. The scores are normalized and then clustered into groups such that models of a same cluster do not show any significant statistical differences in their scores\footnote{Details of the evaluation procedure is outlined in \cite{castro-ferreira20:_2020_bilin_bidir_webnl} and scores are publicly available at \url{https://gerbil-nlg.dice-research.org/gerbil/webnlg2020resultshumaneval}}. When tested on seen categories, we note that our submitted model is competitive with other teams' models. However, on unseen data, although a significant improvement over a simple training of a Transformer, our model shows limitations compared to other participants. A lower rank is systematically observed for both unseen entities and unseen categories.
We assume that a delexicalization step and a much massive pre-training as \cite{devlin-etal-2019-bert, radford2019language} may help to improve our generalization ability on unseen domains. This direction is also to be explored in future work.

\subsection{Pre-Training Objectives}
\input{training/pretraining.tex}

\subsection{Fine-tuning Settings}
\input{training/finetuning.tex}

\subsection{Task Definition}
\input{task_definition/task_definition.tex}
\subsection{Datasets}
\input{data/webnlg_data.tex}

redpajama_set_name: RedPajamaArXiv


text:
\section{Introduction}
Visual Question Answering (VQA) is a challenging problem in computer vision (CV) and natural language processing (NLP) that have gained popularity due to its many-fold benefits ranging from assisting visually impaired users to establishing effective communication with robots \textit{via} intuitive interfaces.\\
\indent The existing works \citep{tan-bansal-2019-lxmert,agrawal2017vqa} on VQA are mainly limited to the English questions, making it challenging to acknowledge progress in foreign languages. Moreover, the current language-vision models do not serve the purpose in the code-mixed setting, where the morphemes, words, phrases of one language are embedded into the other language. Since code-mixing has been a mean of communication in a multi-cultural and multi-lingual society, the next generation of artificial intelligence (AI) agents should be capable to understand the \textbf{M}ultilingual and \textbf{C}ode-\textbf{M}ixed (MCM) questions about the image.\\
\indent In the recent past, the pre-trained \languagevision{} models \cite{Su2020VL-BERT,tan-bansal-2019-lxmert,li2019visualbert,lu2019vilbert} have become the state-of the-arts for solving a variety of CV and NLP problems.
However, the majority of these models are predominantly built for resource-rich languages like English.
Therefore, their abilities to process and answer the MCM questions are limited (\textit{c.f.} Fig. \ref{fig:examples}).\\
\indent To address this, we propose a highly effective and unified VQA method that allows us to extend the existing monolingual \languagevision{} models to multilingual (in \textbf{$6$ different languages}) and code-mixed (in \textbf{$5$ different code-mixed languages}) scenarios. Specifically, we develop a novel knowledge distillation \cite{hinton2015distilling} approach to distill the knowledge from the monolingual \languagevision{} transformer network (teacher model) to multilingual and code-mixed \languagevision{} transformer network (student model). This enables the student model to adapt to any language and code-mixed scenarios.\\
\indent In order to effectively transfer the knowledge from the teacher network to the student network, we introduce multiple distillation objectives which ensure the incremental knowledge extraction from multiple intermediate layers of \languagevision{} transformer model. These objectives are formulated to guide the student model to learn two key characteristics: \textbf{(1)} unified question representation across different languages, and
\textbf{(2)} effective cross-modal (image-question) representation where the key objects in the image are attended irrespective of the language of the questions.
With these characteristics, we are able to build a unified VQA model, which can correctly predict the answers to multilingual and code-mixed questions.\\
\indent Furthermore, to combat the data scarcity, we also create a large-scale ($3.7$M image-question pairs) multilingual and code-mixed VQA dataset. Towards this, we utilize the English VQA1.0 \cite{agrawal2017vqa} dataset and extend it to multiple languages. We also create their code-mixing counterparts by designing the linguistically informed strategy to formulate the code-mixed question by mixing the words/phrases from the English question and the foreign language question. We evaluate our proposed approach on the created dataset and achieve $11.74\%$ average improvement across all the languages over the pre-trained language-vision model.\\
\textbf{Contributions:}
\begin{enumerate}[nolistsep]
 \item We devise a robust {knowledge distillation framework for multilingual and code-mixed VQA} by introducing multiple task-specific objective functions, which distill knowledge from the English pre-trained language-vision model to train and develop equally effective multilingual and code-mixed VQA system.
 \item We create the {large-scale ($3.7$M) multilingual and code-mixed VQA datasets} in
 multiple languages: Hindi (\textit{hi}), Bengali (\textit{bn}), Spanish (\textit{es}), German (\textit{de}), French (\textit{fr}) and code-mixed language pairs: \textit{en-hi}, \textit{en-bn}, \textit{en-fr}, \textit{en-de} and \textit{en-es}. This dataset is publicly available here\footnote{\url{https://www.iitp.ac.in/~ai-nlp-ml/resources.html}}.
\item We demonstrate the effectiveness of our proposed single student model that can correctly predict the answers to the questions of the various language combinations (on {eleven ($11$) different language-vision setups}) including code-mixed setups over state-of-the-art pre-trained language-vision models.
\end{enumerate}

\section{Related Work}
\paragraph{\textbf{Multilingual and Code-Mixing:}}
There is a recent trend in developing methods and resources for various NLP applications involving multilingual and code-mixed languages.
Some of the works include question-answering \citep{raghavi2015answer,gupta-etal-2018-uncovering}, word embedding \cite{chen-cardie:2018:EMNLP,lample2018,word-embedding-for-code-mixed}, code-mixed text generation \cite{pratapa2018language,gonen2018language,gupta-etal-2020-semi}, code-mixed language modelling \cite{winata2018code,gonen2018language}, and other NLP tasks \cite{GUPTA18.486,gupta2016opinion,gupta2016hybrid,gupta2017smpost}.
\paragraph{\textbf{Visual Question Answering:}}
In the literature, various VQA datasets \cite{Silberman:ECCV12,gao2015you,agrawal2017vqa, goyal2017making} have been created to encourage multi-disciplinary research. The popular frameworks for VQA explore attention mechanisms to learn the joint representation of image and question \cite{mcb,mlb,yu2017multi,Kim2018BilinearAN}.
Recently, with the success of Transformer \citep{vaswani2017attention}, \citet{tan-bansal-2019-lxmert} proposed cross-modality framework, \textsc{LXMERT}, for learning the connection between vision and language.
There are other notable works \citep{Su2020VL-BERT,zhou2020unified,li2020oscar}, where the Transformer-based models are pre-trained to learn the joint language-vision representation.
Knowledge distillation has also been used in the literature for the VQA task for the optimal training strategy \cite{mun2018learning}, knowledge transfer from tri-modal to bi-modal \cite{do2019compact}, and the missing modalities \cite{cho2021dealing}. Unlike these, our current work focuses on knowledge transfer from the monolingual pre-trained language-vision model to the multilingual and code-mixed VQA.
\section{Multilingual and Code-Mixed VQA Dataset}
\label{multilingual-dataset}
\paragraph{\textbf{Dataset Creation:}}
We follow the large-scale VQA dataset (VQAv1.0) released by \citet{agrawal2017vqa}.
The VQAv1.0 dataset contains the triplet information in the form of the question, image, and answers. \citet{gupta-etal-2020-unified}
introduced a VQA dataset named MCVQA which comprises of questions in Hindi and Hinglish (i.e. code-mixed English and Hindi). However, the approach to create MCVQA dataset has two major shortcomings: \textbf{(1)} algorithm is not scalable to other languages, and \textbf{(2)} it requires the NLP components (part-of-speech tagger, named entity recognizer,  transliteration, etc.) for the resource-scarce languages, which, themselves are an active research area for the resource-scare languages.

To address these shortcomings, in this work, we create the large-scale ``\textbf{Mu}ltilingual and \textbf{Co}de-mixed \textbf{V}isual \textbf{Q}uestion \textbf{A}nswering'' (\mucovqa{}) dataset which supports the five ($5$) languages (\textit{hi}, \textit{bn}, \textit{es}, \textit{de}, and \textit{fr}) and five ($5$) different code-mixed settings (\textit{en-hi}, \textit{en-bn}, \textit{en-es}, \textit{en-de}, and \textit{en-fr}). To generate the code-mixed questions, we follow the matrix language frame (MLF) theory \cite{myers1997duelling} of code-mixed text.
According to MLF, a code-mixed sentence will have a dominant language (matrix-language) and inserted language (embedded-language). We utilize the Google machine translation to translate the English questions from VQAv1.0 dataset to the foreign language \textit{xx} $\in$ \{\textit{hi}, \textit{bn}, \textit{es}, \textit{de}, \textit{fr}\}. From the parallel questions (\textit{en-xx}), we learn the alignment of English words in the foreign language question. Given a pair of questions from the two languages, we identify the words following \citet{gupta-etal-2020-semi} from the English question and substitute their aligned counterparts (in foreign language question) with the identified English words to synthesize the English embedded code-mixed questions.
Please see \textbf{Appendix} for the implementation details and samples of the \mucovqa{} dataset.

\paragraph{\textbf{Analysis:}}
Similar to the VQAv1.0 dataset, our created \mucovqa{} dataset consists of $248,349$ training and $121,512$ test questions for each of the five different languages and five code-mixed settings.
We perform a qualitative analysis of this dataset by randomly selecting $5,00$ questions, each from \textit{en}, \textit{hi} and corresponding \textit{en-hi}.
We seek annotation help from two bilingual (\textit{en}, \textit{hi}) experts to manually translate and create the code-mixed questions. Towards this, we compute the BLEU \citep{papineni2002bleu}, ROUGE \citep{lin2004rouge} and Translation Error Rate (TER) \citep{snover2006study} considering the manually created code-mixed questions as the gold standard; and the generated code-mixed questions from \mucovqa{} as the candidates. We compute the mean values of the individual scores obtained from both the experts. We found the BLEU: $78.34$, ROUGE-L: $91.13$, and TER: $8.23$, which show the generated code-mixed questions are close to the human formulated code-mixed questions.
The detailed analysis and statistics in terms of code-mixed complexity can be found in the \textbf{Appendix}.

\section{Methodology} \label{sec:methodology}
Our proposed knowledge distillation framework for the VQA model is tailored to predict the answer for multilingual and code-mixed questions.
We utilize LXMERT \cite{tan-bansal-2019-lxmert}, a pre-trained English vision-language model, as the teacher network to train our student network.
Our student network is inspired by the teacher network and has three components, \textit{viz.} \textbf{(1)} \textbf{MCM Question Encoder} that processes and effectively encodes the multilingual and code-mixed questions, \textbf{(2) Image Encoder} which learns the representation of the objects detected in the image, \textbf{(3) Cross-Modality Encoder}, that learns the joint feature representation by applying the cross-attention on the language and image features, and \textbf{(4) Answer Prediction}, which predicts the answer for MCM questions.
\subsection{\textbf{Background}}
\label{lxmert-background}

\paragraph{Transformer Block: } For an input sequence $S^{l}=\{S_1^{l}, S_2^{l}, \ldots, S_{|S|}^{l}\}$ of length $|S|$ (which is the output of the $l^{th}$ transformer block) the $(l+1)^{th}$ transformer block computes the hidden states $S^{l+1}$ as follows:
\begin{equation} \label{eq:transformer-block}
\begin{split}
 \hat{S}_i^{l+1} &= S_i^{l} + \text{MHA}(\text{LayerNorm}(S_i^{l}))\\
  {S}_i^{l+1} &= \hat{S}_i^{l+1} + \text{MLP}(\text{LayerNorm}(\hat{S}_i^{l+1}))\\
\end{split}
\end{equation}
where, $\text{MHA}(.)$ is Multi Head Attention \cite{vaswani2017attention}, $\text{LayerNorm}(.)$ is Layer Normalization \cite{ba2016layer} and $\text{MLP}(.)$ is a feed-forward network.
Based on Eq. \ref{eq:transformer-block}, we define \texttt{Transformer-Block}(.) as a function of input $S^l \in \mathcal{R}^{|S| \times d}$ as follows:
\begin{equation} \label{eq:transformer-block-func}
\begin{split}
  {S}^{l+1} &=  \texttt{Transformer-Block}( {S}^{l})\\
\end{split}
\end{equation}
\subsection{Student Network} \label{sec:student-model}
\paragraph{MCM Question Encoder:} The input question $\mathcal{Q}$ is tokenized using the WordPiece tokenizer~\cite{wu2016google} to form the sequence of tokens $\left\{t_1, t_2, \ldots, t_n\right\}$ with length $n$. We compute the word embedding $w_i$ for the $i^{th}$ token similar to the teacher network LXMERT.
Since, in this work, we deal with multilingual and code-mixed questions; therefore, we utilize Multilingual-BERT (M-BERT) \citep{devlin-etal-2019-bert} model as our language encoder. Multilingual-BERT is a single model pre-trained on the monolingual Wikipedia corpora from $104$ languages. The word embedding sequence $\left\{w_{0=\texttt{[CLS]}}, w_1, \ldots, w_n\right\}$ (with the \texttt{[CLS]} token) is passed to the stack of M-BERT encoders. Each M-BERT encoder consists of the MHA(.) layer followed by point-wise feed-forward network with the residual connection. We obtain the hidden state representation $H^M = \{h_0^{M},  h_1^{M}, \ldots, h_n^{M}\}$ from M-BERT having $M$ layers as follows:
\begin{equation} \label{eq:mbert}
\footnotesize
\begin{split}
 h_0^{1}, \ldots, h_n^{1} &= \text{M-BERT}^{l=1}(w_0, \ldots, w_n)\\
    h_0^{M}, \ldots, h_n^{M} &= \text{M-BERT}^{l=M}(h_0^{M-1}, \ldots, h_n^{M-1})
\end{split}
\end{equation}
For brevity, we will call $\{h_0^{M}, h_1^{M}, \ldots, h_n^{M}\}$ as $H= \{h_0,  h_1, \ldots, h_n\}$ in rest of the paper.

\paragraph{Image Encoder:} Given the input image $\mathcal{I}$, we extract $k$ objects $\{o_1, o_2, \ldots, o_k\}$ from \citet{anderson2018bottom}. For each object $o_j$, we obtain RoI features $r_j \in \mathcal{R}^{d_r}$ and bounding box co-ordinates $b_j \in \mathcal{R}^{d_b}$. We follow the object-relationship encoder from \citet{tan-bansal-2019-lxmert} to obtain the image representation. We first project RoI and co-ordinates via a feed-forward network to obtain $f_j$ and $p_j$, respectively. Then we obtain the object feature for the object $o_j$ as $u_j = (f_j + p_j)/{2}\in \mathcal{R}^{d}$ .
With $k$ objects in the image, we obtain the object feature matrix $U^0 \in \mathcal{R}^{k \times d}$. We employ the stack of \tblock{} (\textit{c.f.} Eq . \ref{eq:transformer-block-func}) to encode the image. For the first \tblock{}, we fed the object feature matrix $U^0$ and obtain
the hidden state representations $u_1^{1}, \ldots, u_k^{1}$. Subsequently, we obtain the final image representation $U= U^N \in \mathcal{R}^{k \times d}$ from the last layer ($N$) of \tblock{} as follows:
\begin{equation}
\footnotesize
u_1, \ldots, u_k =\tblock{}(U^{N-1})
\end{equation}
\paragraph{Cross-Modality Encoder:}
Given the MCM question representation $H \in \mathcal{R}^{n \times d}$ and image representation $U \in \mathcal{R}^{k \times d}$, similar to \citet{tan-bansal-2019-lxmert}, we aim to compute the cross-modal representations using the layers of \tblock{}. For a given layer $l$, the cross-modality encoder consists of two cross-attention layers (one from question to image another from image to question) and two \tblock{} for each modality. Cross-attention layer $\crossatt{}(.)$ takes the query vector $x^q$ of the representation $x$ from one of the modals and compute the attention weight $\alpha_j = softmax(x^q.y_j^k)$ with the key vectors= $y_j^k$ from the other modality. Thereafter, it computes the final cross-modal representation $\overline{x}= \sum \alpha_j y_j^v$ as the weighted average of the set of value vectors $\{y^v\}$. For the cross-modal representation $\overline{H}^l \in \mathcal{R}^{n \times d}$ from the $l^{th}$ layer of the question, we apply the \crossatt{} followed by the \tblock{} operation as follows:
\begin{equation} \label{eq:cross-modal-question}
\footnotesize
\begin{split}
  \widetilde{h}_i^{l}&= \crossatt{}(h_i^{l-1}, [u_1^{l-1}, u_2^{l-1}, \ldots, u_k^{l-1}] )\\
  \widetilde{H}^l & = [\widetilde{h}_0^{l}, \widetilde{h}_1^{l}, \ldots, \widetilde{h}_n^{l} ] \in \mathcal{R}^{n \times d}\\
 \overline{H}^l &= \tblock{}(\widetilde{H}^{l})
\end{split}
\end{equation}
Similarly, the cross-modal representation $ \overline{U}^l$ for the image considering the question as another modal is computed.
We use the $L$ layers of cross-modal encoders to encode the cross-modal representation.
\paragraph{Answer Prediction: } To predict the answer for the multilingual question, we take the output of question from the last ($L^{th}$) cross-modal encoders. We use the \texttt{[CLS]} token representation $\overline{h}_{\texttt{[CLS]}}^{L} \in \mathcal{R}^{d}$ and predict the answer as follows:
\begin{equation} \label{eq:answer-predict}
\footnotesize
\begin{split}
  \overline{P} &= gelu(\mathbf{W_P}\overline{h}_{\texttt{[CLS]}}^{L} + c_{P})\\
  p({\mathcal{A}_{i}}|\mathbf{\mathcal{X}}; \theta^{S}) &= \sigma(\mathbf{W}_{i} \overline{P}+ c_{i})
\end{split}
\end{equation}
where, $\mathbf{W_P} \in \mathcal{R}^{2d \times d}$ is the weight matrix and $c_{P} \in \mathcal{R}^{2d}$ is the bias vector. $\sigma$ denotes the sigmoid function. $W_i$ and $c_i$ are the $i^{th}$ entry of weight matrix $W \in \mathcal{R}^{d \times |\mathcal{A}|}$ and bias vector $c \in \mathcal{R}^{ |\mathcal{A}|}$.
$\overline{h}_{\texttt{[CLS]}}^{L} \in \mathcal{R}^{d}$ is the hidden state representation of \texttt{[CLS]} token obtained from cross-modality encoder. $|\mathcal{A}|$ is the length of the answer vocabulary. $\mathcal{X}$ is the set of input $\{\mathcal{Q}, \mathcal{I}\}$. $p({\mathcal{A}_{i}}|\mathbf{\mathcal{X}}; \theta^{S})$ is the probability of the $i^{th}$ answer from answer vocabulary $\mathcal{A}$.
\subsection{Distillation Objectives}
In our knowledge-distillation framework, we propose multiple objectives to transfer the knowledge from the monolingual Teacher network (with $\theta^T$ parameters) to the MCM Student network (with $\theta^S$ parameters):
\paragraph{Objective 1 - \texttt{CLS} Token Distillation:} The \texttt{[CLS]} token embedding learned at the cross-modality encoder represents the semantics of the monolingual question-image pair in teacher network and MCM question-image pair in student network. We argue that it should learn a similar representation to correctly predict the answer irrespective of the language. Towards this, we compute the \texttt{[CLS]} token loss by computing the Mean Squared Error (MSE) between the vector representation learned at the Cross-modality Encoder in the teacher network and student network.
\begin{equation} \label{eq:clas-loss}
\footnotesize
\begin{split}
\mathcal{L}_{CLS} = \sum_{i=1}^{i=|L|} \sum_{j=1}^{j=|MH|} \mathbf{MSE}(\overline{h}_{(i,j,[CLS])}^{T}, \overline{h}_{(i,j,[CLS])}^{S})
\end{split}
\end{equation}
where, $\overline{h}_{(i,j,[CLS])}^{T} \in \mathcal{R}^{d}$ and $\overline{h}_{(i,j,[CLS])}^{S} \in \mathcal{R}^{d}$ are the representation of the \texttt{[CLS]} token obtained from the $i^{th}$ cross-modal encoder layer under $j^{th}$ attention head from teacher and student network, respectively. $|MH|$ is the number of attention head in the \tblock{}.

\paragraph{Objective 2 - Object Attention Distillation:} The answer to a given question is defined by the object detected in the image. It is to be noted that the answer to a question is independent of the language. We argue that in order to correctly predict the answer to MCM questions, the student network should attend the same object as the teacher network. This helps in aligning the question representation across different languages to the object representation and thus assists towards learning the effective language-agnostic cross-modal representation of the question-image pair. Towards this, we compute the object attention loss ($\mathcal{L}_{object}$), which measures the MSE between the raw score vectors $z \in \mathcal{R}^{k}$ (obtained using the dot product between \texttt{[CLS]} token's query vector and set of object's key vector) learned at the Cross-modality Encoder in the teacher network and student network.
\begin{equation} \label{eq:object-loss}
\footnotesize
\begin{split}
\mathcal{L}_{object} = \sum_{i=1}^{i=|L|} \sum_{j=1}^{j=|MH|} \mathbf{MSE}(z_{(i,j)}^{T}, z_{(i,j)}^{S})
\end{split}
\end{equation}
where, $z_{(i,j)}^{T} \in \mathcal{R}^{k}$ and $z_{(i,j)}^{S} \in \mathcal{R}^{k}$ are the vector raw scores obtained from the $i^{th}$ layer under the $j^{th}$ attention head from Teacher and Student network, respectively.

\paragraph{Objective 3 - Prediction Distillation:}  In addition to imitating the behaviors of intermediate layers, we
also use the knowledge distillation to mimic the predictions of teacher network. Specifically, we penalize the binary cross-entropy loss between the answer probabilities obtained from the teacher and student network.
\begin{equation} \label{eq:prediction}
\footnotesize
\begin{split}
\mathcal{L}_{pred} &= -\sum_{i=1}^{i=|\mathcal{A}|} p(\mathcal{A}_i|\mathcal{X}; \theta^T) log(p(\mathcal{A}_i|\mathcal{X}; \theta^S)) + \\
& (1- p(\mathcal{A}_i|\mathcal{X}; \theta^T)) log(1- p(\mathcal{A}_i|\mathcal{X}; \theta^S))
\end{split}
\end{equation}

\paragraph{Objective 4 - Negative Log-likelihood Loss:}
We also penalize the binary cross-entropy loss between the gold answer probability $y_i$ and model's predicted probability $p(\mathcal{A}_i|\mathcal{X}; \theta^S)$ obtained from the student network.
\vspace{-1em}
\begin{equation} \label{eq:nll}
\footnotesize
\begin{split}
\mathcal{L}_{nll} &= -\sum_{i=1}^{i=|\mathcal{A}|} y_ilog(p(\mathcal{A}_i|\mathcal{X}; \theta^S)) +  \\
&(1-y_i)log(1- p(\mathcal{A}_i|\mathcal{X}; \theta^S))
\end{split}
\end{equation}

\subsection{Learning}
To apply the knowledge distillation, first we need to
train our Teacher network,
having $\theta^{T}$ parameters with English question from the VQAv1.0 dataset. Thereafter, the Teacher network's parameters are frozen and the Student network is trained with the following objective function:
\begin{equation} \label{eq:student-loss}
\begin{split}
\mathcal{L} = \mathcal{L}_{\texttt{CLS}} + \mathcal{L}_{\texttt{object}} + \mathcal{L}_{\texttt{pred}} + \mathcal{L}_{\texttt{nll}}
\end{split}
\end{equation}
During training, the Teacher network is fed with the English question and the corresponding image, and the Student network is fed with multilingual and code-mixed questions (one language at a time) and the corresponding image.

\section{Dataset and Experiments} \label{sec:experiments}
\subsection{Datasets} We evaluate our proposed knowledge distillation framework on the \mucovqa{} dataset having eleven different language setups, and the MCVQA dataset \cite{gupta-etal-2020-unified} that consists of \textit{en}, \textit{hi}, \textit{en-hi} language setups. We train the Student network with the training dataset from all these languages.
We take out $5\%$ of the training dataset as the validation dataset for evaluating and selecting the best Student model. The best Student model is used to evaluate the performance of the \mucovqa{} test dataset in all the language setups. For evaluation, we follow the accuracy metric as defined in \citet{agrawal2017vqa}.

\subsection{Implementation Details}
We use the pre-trained Multilingual BERT\footnote{\url{https://github.com/google-research/bert/blob/master/multilingual.md}} having $12$ encoder layers, each having $12$ attention heads and a hidden dimension of $768$ for each token. In our proposed knowledge distillation framework, we train the model on \mucovqa{} training dataset for $16$ epochs.
Since the input image remains the same for the LXMERT model and our proposed model, we initialize our image encoder weights with the LXMERT object relationship encoders' weights.

We set the maximum question length to $20$ words. The numbers of objects extracted from the image is $k=36$ and the dimension of bounding box coordinates and RoI features are $d_{b}=4$ and $d_r=2048$, respectively. For the Teacher network, the language encoder has $M=9$ layers, the image encoder has $N=5$ layers and the cross-modality encoder has the $L=5$ layers. Similarly in the Student network, the values of these layers are $M=12, N=5, L=5$. During training, we fine-tune the top $4$ M-BERT encoders and the top $2$ image encoders. We learn the cross attention layer from scratch to align the multilingual and vision embeddings. For the \texttt{CLS} token distillation, we set the layers $i \in \{1, 4\}$ and attention head $j \in \{1, 4, 5\}$.
Optimal values of the hyperparameters are chosen based on the model performance on the development set of \mucovqa{} dataset.

\subsection{Baselines} \label{sec:baselines}
We compare the performance of the proposed network with the following baseline models.\\
\textbf{(1)} \textbf{LXMERT}: We train the individual LXMERT model on the training dataset of each language from \mucovqa{} dataset and evaluate the performance on the respective test dataset. \\
\textbf{(2)} \textbf{Joint LXMERT}: We train the single LXMERT model on all the training datasets of each language from \mucovqa{} dataset and evaluate the performance on the respective test dataset. \\
\textbf{(3)} \textbf{Joint LXMERT+ M-BERT}: This baseline is similar to the Joint LXMERT but the monolingual language encoder is replaced with a multilingual M-BERT encoder. \\
\textbf{(4)} \textbf{VL-BERT} \cite{Su2020VL-BERT}:
We also compare the performance of our proposed model with the VL-BERT base model (\texttt{vl-bert-base-e2e.model}). We train separate VL-BERT model on the training dataset of each language from the \mucovqa{} dataset and evaluate the performance on the respective test dataset. \\
\textbf{(5)} \textbf{VisualBERT} \cite{li2019visualbert}:
Similar to the LXMERT, we also compare the performance of our proposed network on the \mucovqa{} dataset with the VisualBERT monolingual model.

\begin{table*}[]
\resizebox{\textwidth}{!}{%
\begin{tabular}{l|c|cccccccccccccc}
\hline
 & \textbf{Models} & \textbf{\textit{bn}} & \textbf{\textit{en-bn}} & \textbf{\textit{de}} & \textbf{\textit{en-de}} & \textbf{\textit{es}} & \textbf{\textit{en-es}} & \textbf{\textit{fr}} & \textbf{\textit{en-fr}} & \textbf{\textit{hi}} & \textbf{\textit{en-hi}} & \textbf{\textit{en}} & \textbf{\textit{Average}} \\ \hline \hline
\multirow{6}{*}{\rotatebox[origin=c]{90}{\textbf{Monolingual}}} &
 \begin{tabular}[c]{@{}c@{}}LXMERT\\ \cite{tan-bansal-2019-lxmert}\end{tabular}  & 60.74 & 64.95 & 67.95 & 70.52 & 68.66 & \textbf{71.27} & 68.43 & 71.12 & 59.83 & 69.95 & \textbf{73.57} & 67.90 \\
&\begin{tabular}[c]{@{}c@{}}VL-BERT\\ \cite{Su2020VL-BERT}\end{tabular} & 58.53 & 61.30 & 64.29 & 65.33 & 64.79 & 66.09 & 64.72 & 65.84 & 59.40 & 65.28 & 67.30 & 63.89 \\
&\begin{tabular}[c]{@{}c@{}}VisualBERT\\ \cite{lu2019vilbert}\end{tabular}
&61.45&	64.20 &	66.74&	67.49&	67.31&	67.42&	66.35&	67.21&	59.68&	63.67&	68.12&	65.42 \\ \hline

\multirow{8}{*}{\rotatebox[origin=c]{90}{\textbf{Multilingual}}} &\begin{tabular}[c]{@{}c@{}}Joint LXMERT\\ \cite{tan-bansal-2019-lxmert}\end{tabular} & 48.44 & 60.07 & 58.02 & 62.79 & 58.41 & 63.07 & 58.34 & 62.96 & 50.28 & 61.52 & 65.41 & 59.02 \\
&Joint LXMERT+ M-BERT & 55.68 & 56.89 & 57.73 & 58.01 & 57.87 & 58.34 & 57.45 & 57.82 & 56.22 & 57.18 & 58.78 & 57.45
\\ \cline{2-14}
&\textbf{Proposed Approach} & {\textbf{69.62}} & \textbf{70.19} & \textbf{70.89} & \textbf{70.80} & \textbf{71.14} & 71.11 & \textbf{70.93} & \textbf{71.13} & \textbf{70.23} & \textbf{70.78} & 71.66 & \textbf{70.76} \\  \cline{2-14}
&\quad $-\mathcal{L}_{CLS}$ & 67.95 & 68.50 & 69.18 & 69.06 & 69.45 & 69.59 & 69.23 & 69.42 & 68.55 & 69.05 & 69.91 & 69.08 \\
&\quad $-\mathcal{L}_{object}$ & 66.02 & 66.56 & 67.30 & 67.32 & 67.60 & 67.74 & 67.32 & 67.56 & 66.66 & 67.13 & 68.80 & 67.27 \\
&\quad $-\mathcal{L}_{pred}$ & 68.17 & 68.77 & 69.40 & 69.32 & 69.65 & 69.88 & 69.42 & 69.62 & 68.77 & 69.30 & 70.11 & 69.31 \\
&\quad $-\mathcal{L}_{nll}$ & 68.17 & 68.83 & 69.50 & 69.41 & 69.70 & 69.88 & 69.48 & 69.47 & 68.80 & 69.35 & 70.28 & 69.35 \\ \hline
\end{tabular}%
}
\caption{Performance comparison between the state-of-the-art baselines and our proposed model on the \mucovqa{} dataset. All the numbers are shown in \% and denote the overall accuracy.
}
\label{tab:main-results}
\end{table*}

\begin{table}[h]
\resizebox{\linewidth}{!}{%
\begin{tabular}{ccccc}
\hline
\textbf{Models} & \textbf{\textit{en}} & \textbf{\textit{hi}} & \textbf{\textit{en-hi}} & \textbf{\textit{Average}} \\ \hline
LXMERT \cite{tan-bansal-2019-lxmert} & \textbf{73.02} & 63.33 & 68.77 & 68.37\\
VL-BERT \cite{Su2020VL-BERT} &67.28& 59.32 & 63.28 & 63.29  \\
VisualBERT \cite{li2019visualbert} &68.04& 59.69 & 63.62 & 63.78  \\
\citet{gupta-etal-2020-unified} & 65.37 & 64.51 & 64.69 &  64.85\\ \hline
\textbf{Proposed Approach} & 71.37 & \textbf{69.94 }& \textbf{69.47} & \textbf{70.26}\\ \hline
\end{tabular}%
}
\caption{Performance comparison of different models on the MCVQA dataset.}
\label{tab:mcvqa-results}
\end{table}

\begin{table}[]
\resizebox{\linewidth}{!}{%
\begin{tabular}{ccccc}
\hline
\textbf{Language} & \textbf{Number} & \textbf{Other} & \textbf{Yes/No} & \textbf{Overall} \\ \hline \hline
\textit{\textbf{en}} & 51.15 & 64.56 & 88.02 &  71.66 \\
\textit{\textbf{bn}} & 50.62 & 62.16 & 85.97 &  69.62 \\
\textit{\textbf{en-bn}} & 50.78 & 62.89 & 86.45 & 70.19 \\
\textit{\textbf{de}} & 50.86 & 63.50 & 87.49 & 70.89 \\
\textit{\textbf{en-de}} & 50.84 & 63.34 & 87.45 &  70.80\\
\textit{\textbf{es}} & 50.93 & 63.95 & 87.54 & 71.14 \\
\textit{\textbf{en-es}} & 50.94 & 64.19 & 87.68 & 71.11 \\
\textit{\textbf{fr}} & 50.95 & 63.47 & 87.61 &70.93   \\
\textit{\textbf{en-fr}} & 51.01 & 63.84 & 87.58 & 71.13 \\
\textit{\textbf{hi}} & 50.72 & 62.57 & 87.01 &  70.23\\
\textit{\textbf{en-hi}} & 50.95 & 63.30 & 87.43 &  70.78\\ \hline \hline
\end{tabular}%
}
\caption{Performance of our proposed model on different answer types across all the language setups in \mucovqa{} dataset}
\label{tab:lang-results}
\end{table}

\subsection{Results} \label{sec:results}
We report the performance of the baseline models and our proposed model on \mucovqa{} dataset in Table \ref{tab:main-results}. We also reported the answer-type wise results on \mucovqa{} dataset in Table \ref{tab:lang-results}.
Our proposed model achieves $70.76$ overall accuracy and outperforms the best monolingual and multilingual baselines with significant improvements of $2.86$ and $11.74$, respectively. Our proposed approach also outperforms the state-of-the-art model on MCVQA dataset (\textit{c.f.} Table \ref{tab:mcvqa-results}) with considerable performance improvement of $5.07\%$.
We could not observe a similar improvement on \textit{en} language, because the LXMERT teacher model (\textit{en}) is already pre-trained with the English VQA dataset.

It is to be noted that each monolingual model is trained separately with the respective language dataset and has a different model for each language setup.
The results conclude two important claims: \textbf{(1)} effectiveness of knowledge distillation approach to handle MCM questions, and \textbf{(2)} scalability of our proposed single unified VQA model that can deal with questions from all the languages and their code-mixed setups.\\
\indent We also perform the ablation study (\textit{c.f.} Table \ref{tab:main-results}) on different distillation objective functions.
The results show that Object Attention Distillation ($\mathcal{L}_{object}$) is the most contributing objective function, removal of which leads to the $3.49\%$ decrements in the overall average accuracy. We also observe the importance of the \texttt{CLS} Token Distillation ($\mathcal{L}_{\texttt{CLS}}$). This is the key loss function responsible to align the same multilingual and code-mixed questions in the vector space, and removing it leads to $1.68\%$ decrements in overall average accuracy.
Similarly, we observe $1.45\%$ and $1.41\%$ performance drops after the removal of $\mathcal{L}_{pred}$ and $\mathcal{L}_{nll}$ objective functions, respectively. The observed improvements over the multilingual baselines are statistically significant as $p< 0.05$ for the t-test using \citet{statistical-test}.
Please see the \textbf{Appendix} for additional results.
 \begin{figure}[t]
    \centering
    \includegraphics[ width=\linewidth]{May15/plot1-main-paper-final.pdf}
    \caption{Performance comparison of different models for question understanding by varying the partial question as input to the model.}
    \label{fig:question-understanding-incomplete-main}
\end{figure}
\begin{figure}[]
  \subfloat[\centering \textbf{Joint LXMERT}]
  {{\includegraphics[width=0.45\linewidth]{images/JointLxmert_embeddings_final.pdf} }}%
    \quad
    \subfloat[\centering \textbf{Proposed Approach}]
    {{\includegraphics[width=0.45\linewidth]{images/Proposed_embeddings_final.pdf} }}%
     \vspace{5pt}
\captionof{figure}{t-SNE visualization for MCM questions in all eleven language setups. For proposed approach (b), we observe that the question representations of the same questions (shown in the same color) in different languages are very close in vector space unlike the Joint LXMERT model (a).}
\label{question-reps}
 \end{figure}

\begin{figure}[t]
 \begin{minipage}{.1\textwidth}
  \centering
 {{\includegraphics[width=\linewidth]{images/COCO_val2014_000000155380.jpg} }}%
 \end{minipage}
    \quad
    \begin{minipage}{.35\textwidth}
  {{\includegraphics[width=\linewidth]{May15/heatmap1-main-paper-crop.pdf}}}%
  \end{minipage}
\captionof{figure}{Heatmap of the learned attention weight for objects in the image from our proposed model. The proposed model is able to focus on the same object and correctly predict the answer irrespective of the language of the question. x-axis shows the heads of self-attention.}
\label{fig:question-object-alignment-main}
 \end{figure}

 \begin{figure}[t]
    \centering
    \includegraphics[ width=\linewidth]{May15/rand-questions-crop.pdf}
    \caption{Performance comparison of the proposed model with shuffled questions and original questions}
    \label{fig:question-shuffle}
\end{figure}

\begin{figure}[]
    \centering
    \includegraphics[ width=\linewidth]{May15/zero-crop.pdf}
    \caption{Zero-shot performance comparison of the proposed model on the different languages}
    \label{fig:zero-shot}
\end{figure}

   \begin{figure*}[h]
      \centering
      \includegraphics[width=0.95\linewidth]{error_analysis/error-analysis-new.pdf}
      \caption{Examples from the various type of errors committed by our proposed model}
      \label{fig:vqa-error-analysis}
  \end{figure*}

\subsection{Discussion and Analysis} \label{sec:discussion}
\paragraph{Behavior Analysis:}
We analyze the behavior of our proposed VQA model along the following dimensions:\\
\textbf{(a) Question Understanding}:
Motivated from \citet{agrawal2016analyzing}, we analyze the performance of the model as a function of partial question length to establish the fact that the proposed model is more sensitive to MCM questions as compared to other pre-trained models.
To examine this, we fed the LXMERT (monolingual), Joint-LXMERT (multilingual), and the proposed model (multilingual) with partial questions in the range of $20$ to $100$\% in an incremental manner. We observe (\textit{c.f.} bar chart in Fig. \ref{fig:question-understanding-incomplete-main}) that our proposed model does not jump to quick conclusions by looking at partial questions as the overall accuracy is comparatively low for the proposed model for the incomplete questions. However, with full questions, the accuracies are high for the proposed model indicating that the model is sensitive to questions in different languages.\\
\indent Furthermore, we also analyze what percentage of answers do not change when the partial questions are provided as input to the model.
We can observe from the line chart of Fig. \ref{fig:question-understanding-incomplete-main} that our proposed model is capable to change the answers when more question words are received as input to the model, unlike the LXMERT and Joint LXMERT model where the answers remain the same for around $50\%$ of the questions.
Additionally, to assess the role of syntax and semantics of the multilingual input questions, we analyze the performance of the system by feeding the randomly shuffled questions in Fig \ref{fig:question-shuffle}. The results show that our model is capable to understand the question semantics.\\
\textbf{(b) Alignment}:
We also analyze the alignment of the learned MCM question representation from our MCM Question Encoder. Towards this, we project the question representation (\texttt{[CLS]}) of the same question asked in different MCM settings using the t-SNE visualization \cite{van2008visualizing} in Fig \ref{question-reps}. The plot shows that the question representations learned from the Joint LXMERT model are scattered in the vector space. In contrast, our proposed model learns the question representations which are very close in the vector space, indicating the capability of the model to learn the language-agnostic question representations, which help the model to correctly predict the answer of the MCM questions.

In addition, we also analyze the cross-modal alignment learned from our proposed model. Towards this, we plot the attention heatmap (\textit{c.f.} Fig \ref{fig:question-object-alignment-main}) from the cross-modal encoder (\crossatt{}). We analyze that our proposed model is able to effectively learn the language-agnostic cross-modal representation, where the key objects from the images are attended to predict the correct answer for MCM questions. We also show (in \textbf{Appendix}) that the cross-modal representation learned from the proposed model is tightly coupled with the image and question as the attention to objects get change when the different questions are asked from the same image.
Overall this analysis confirms that our model is not myopic to images and MCM questions to predict the answers.
\\
\textbf{(c) Zero-shot Capability}: We also assess the zero-shot capability of our proposed model. Towards this, we perform the experiments on the six more languages, \textit{viz.} Arabic (\textit{ar}), Italian (\textit{it}), Russian (\textit{ru}), Urdu (\textit{ur}), Polish (\textit{pl}), and Portuguese (\textit{pt}). We evaluate the performance of our proposed model in zero-shot manner on the $500$ questions translated into the respective languages (using Google translation). We compare the performance (\textit{c.f.} Fig \ref{fig:zero-shot}) with the multilingual Joint LXMERT model.
The proposed model achieves better overall accuracy compared to the Joint LXMERT model.
This demonstrates the capability of our model on the unseen languages, which eventually confirms that the proposed distillation objectives have guided the student to learn the robust cross-modal representations.
Please see the \textbf{Appendix} for the detailed qualitative analysis.
\vspace{-0.5 em}
\paragraph{Error Analysis:}
We categorize the following major sources of errors by sampling $200$ incorrectly predicted answers:\\
\textbf{(a)} \textbf{Answer Specificity and Ambiguity (E1)}: This type of error occurs when the objects in the image can be interpreted in multiple ways based on their visual surroundings.  In those cases, our model sometimes predicts the incorrect but semantically similar to the ground truth (GT) answer. For example, $\mathbf{Q1}$ in Fig.  \ref{fig:vqa-error-analysis}, the question is \textit{``What is the man holding"}. Our model predicts the `\textit{bird}' as the answer for all languages of the questions. However, the ground truth answer is `\textit{Turkey}' which is more specific and semantically similar.\\
\textbf{(b)} \textbf{Object Counting (E2)}: We observe that our proposed model sometime predicts the incorrect answer for the counting type questions. The example is shown as $\mathbf{Q2}$ of Fig  \ref{fig:vqa-error-analysis}.\\
\textbf{(c)} \textbf{Character Recognition (E3)}:
This type of error occurs when the answer to the MCM questions can only be predicted by recognizing the characters from the images. The example is shown in Fig. \ref{fig:vqa-error-analysis} ($\mathbf{Q3}$), where the GT answer is `$30$' (speed limit) but the model predicts the incorrect answer `$25$' because it could not recognize the character written in the image.\\
\textbf{(d)} \textbf{Spatial Interpretation (E4)}:
Such errors occur when the model could not correctly interpret the spatial information in the image. The example is shown in Fig. \ref{fig:vqa-error-analysis} ($\mathbf{Q4}$), where the model predicted the `\textit{pillow}' as the answer instead `\textit{broom}'.\\
\textbf{(e)} \textbf{Answer Reasoning (E5)}:
This type of error occurs for the question, which requires understanding the causal relationship or in-depth reasoning to correctly predict the answer. We show the example (Fig. \ref{fig:vqa-error-analysis} ($\mathbf{Q5}$)), where to infer the age of the boy, the system has to establish the fact that \textit{number of candles on the cake can determine the age}.
There are some other errors caused by parallel question alignment and translation of the questions. We found the error \textbf{E5} contributes to the maximum of $26.5\%$, \textbf{E1}: $23.5\%$, \textbf{E3}: $21\%$, \textbf{E2}: $16\%$, \textbf{E4}: $9\%$ and other types of error contributes to $4\%$ of the total errors.
\section{Conclusion} \label{sec:conclusion}
In this paper, we have proposed a unified framework for multilingual and code-mixed VQA by distilling the knowledge from the monolingual language-vision pre-trained LXMERT model. To fully utilize the rich information from the question, image, and cross-modal encoders, we devise effective distillation objectives to encourages the student model to learn from the teacher through a multi-layer distillation process. To train and evaluate the proposed approach, we have created a large-scale \mucovqa{} dataset supporting eleven different MCM settings. Extensive experiments over the \mucovqa{} and MCVQA datasets demonstrate the effectiveness of our proposed approach.

\section*{Ethical Declaration}
All the datasets used in this paper are publicly available. The dataset used in this paper is used only for the purpose of academic research. There are no ethical concerns associated with the research carried out here.
\section*{Acknowledgement}
Asif Ekbal acknowledges the Young Faculty Research Fellowship (YFRF), supported by Visvesvaraya PhD scheme for Electronics and IT, Ministry of Electronics and Information Technology (MeitY), Government of India, being implemented by Digital India Corporation (formerly Media Lab Asia).