Lost in the Mix: Evaluating LLM Understanding of Code-Switched Text
Code-switching (CSW) is the act of alternating between two or more languages
within a single discourse. This phenomenon is widespread in multilingual
communities, and increasingly prevalent in online content, where users
naturally mix languages in everyday communication. As a result, Large Language
Models (LLMs), now central to content processing and generation, are frequently
exposed to code-switched inputs. Given their widespread use, it is crucial to
understand how LLMs process and reason about such mixed-language text. This
paper presents a systematic evaluation of LLM comprehension under
code-switching by generating CSW variants of established reasoning and
comprehension benchmarks. While degradation is evident when foreign tokens
disrupt English textx2013even under linguistic
constraintsx2013embedding English into other languages often
improves comprehension. Though prompting yields mixed results, fine-tuning
offers a more stable path to degradation mitigation.
685b863bd2ee4fac76521e92
Amr Mohamed
claimed_verified
2025-06-25T08:07:19.266Z
655efd24afee0e00788bb589
https://cdn-avatars.huggingface.co/v1/production/uploads/655efd24afee0e00788bb589/22guLxIWNybbJR3jI-c4w.jpeg
Amr Mohamed
user
amr-mohamed
685b863bd2ee4fac76521e93
Yang Zhang
claimed_verified
2025-06-25T12:56:26.602Z
64f1f92eefacc7da583a9e22
/avatars/18abbf9e31ca916f8d9a4495639a1329.svg
Yang ZHANG
user
yangzhang33
685b863bd2ee4fac76521e94
Michalis Vazirgiannis
685b863bd2ee4fac76521e95
Guokan Shang
claimed_verified
2025-06-25T08:07:16.772Z
6087e598e2b7cc3a117b0dc5
https://cdn-avatars.huggingface.co/v1/production/uploads/6087e598e2b7cc3a117b0dc5/Ctz_W-uo1gOQRBHXalD1P.png
Guokan Shang
user
guokan-shang
Lost in the Mix: Evaluating LLM Understanding of Code-Switched Text
Code-switching (CSW) is the act of alternating between two or more languages
within a single discourse. This phenomenon is widespread in multilingual
communities, and increasingly prevalent in online content, where users
naturally mix languages in everyday communication. As a result, Large Language
Models (LLMs), now central to content processing and generation, are frequently
exposed to code-switched inputs. Given their widespread use, it is crucial to
understand how LLMs process and reason about such mixed-language text. This
paper presents a systematic evaluation of LLM comprehension under
code-switching by generating CSW variants of established reasoning and
comprehension benchmarks. While degradation is evident when foreign tokens
disrupt English textx2013even under linguistic
constraintsx2013embedding English into other languages often
improves comprehension. Though prompting yields mixed results, fine-tuning
offers a more stable path to degradation mitigation.
LLMs' comprehension and reasoning skills are evaluated under code-switching conditions, revealing that embedding English into other languages can improve understanding, while prompts and fine-tuning affect degradation mitigation differently.
Large Language Models code-switching CSW reasoning benchmarks comprehension benchmarks foreign tokens embedding fine-tuning