Concept Space Alignment in Multilingual LLMs
<|reference_start|>Concept Space Alignment in Multilingual LLMs: Multilingual large language models (LLMs) seem to generalize somewhat across languages. We hypothesize this is a result of implicit vector space alignment. Evaluating such alignment, we see that larger models exhibit very high-quality linear alignments between corresponding concepts in different languages. Our experiments show that multilingual LLMs suffer from two familiar weaknesses: generalization works best for languages with similar typology, and for abstract concepts. For some models, e.g., the Llama-2 family of models, prompt-based embeddings align better than word embeddings, but the projections are less linear -- an observation that holds across almost all model families, indicating that some of the implicitly learned alignments are broken somewhat by prompt-based methods.<|reference_end|>
@article{peng2024concept,
    title={Concept Space Alignment in Multilingual LLMs},
    author={Qiwei Peng and Anders S{o}gaard},
    journal={arXiv preprint arXiv:2410.01079},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.01079},
    primaryClass={cs.CL}
}