Exploring Alignment in Shared Cross-lingual Spaces
<|reference_start|>Exploring Alignment in Shared Cross-lingual Spaces: Despite their remarkable ability to capture linguistic nuances across diverse languages, questions persist regarding the degree of alignment between languages in multilingual embeddings. Drawing inspiration from research on high-dimensional representations in neural language models, we employ clustering to uncover latent concepts within multilingual models. Our analysis focuses on quantifying the \textit{alignment} and \textit{overlap} of these concepts across various languages within the latent space. To this end, we introduce two metrics \CA{} and \CO{} aimed at quantifying these aspects, enabling a deeper exploration of multilingual embeddings. Our study encompasses three multilingual models (\texttt{mT5}, \texttt{mBERT}, and \texttt{XLM-R}) and three downstream tasks (Machine Translation, Named Entity Recognition, and Sentiment Analysis). Key findings from our analysis include: i) deeper layers in the network demonstrate increased cross-lingual \textit{alignment} due to the presence of language-agnostic concepts, ii) fine-tuning of the models enhances \textit{alignment} within the latent space, and iii) such task-specific calibration helps in explaining the emergence of zero-shot capabilities in the models.\footnote{The code is available at \url{https://github.com/baselmousi/multilingual-latent-concepts}}<|reference_end|>
@article{mousi2024exploring,
    title={Exploring Alignment in Shared Cross-lingual Spaces},
    author={Basel Mousi and Nadir Durrani and Fahim Dalvi and Majd Hawasly and
  Ahmed Abdelali},
    journal={arXiv preprint arXiv:2405.14535},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.14535},
    primaryClass={cs.CL cs.AI}
}