title: CS-Sum: A Benchmark for Code-Switching Dialogue Summarization and the
    Limits of Large Language Models
summary: Code-switching (CS) poses a significant challenge for Large Language Models
  (LLMs), yet its comprehensibility remains underexplored in LLMs. We introduce
  CS-Sum, to evaluate the comprehensibility of CS by the LLMs through CS dialogue
  to English summarization. CS-Sum is the first benchmark for CS dialogue
  summarization across Mandarin-English (EN-ZH), Tamil-English (EN-TA), and
  Malay-English (EN-MS), with 900-1300 human-annotated dialogues per language
  pair. Evaluating ten LLMs, including open and closed-source models, we analyze
  performance across few-shot, translate-summarize, and fine-tuning (LoRA, QLoRA
  on synthetic data) approaches. Our findings show that though the scores on
  automated metrics are high, LLMs make subtle mistakes that alter the complete
  meaning of the dialogue. To this end, we introduce 3 most common type of errors
  that LLMs make when handling CS input. Error rates vary across CS pairs and
  LLMs, with some LLMs showing more frequent errors on certain language pairs,
  underscoring the need for specialized training on code-switched data.
paper_authors._id: 682d4ccc3b5f51f4218e12b4
paper_authors.name: Sathya Krishnan Suresh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-21T10:18:55.833Z
paper_authors._id: 62f0e457bc8201db9ef47f89
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f0e457bc8201db9ef47f89/zOhDptwZpDGaugKCBOWB2.jpeg
paper_authors.fullname: Sathya Krishnan
paper_authors.type: user
paper_authors.user: SkAndMl
paper_authors._id: 682d4ccc3b5f51f4218e12b5
paper_authors.name: Tanmay Surana
paper_authors._id: 682d4ccc3b5f51f4218e12b6
paper_authors.name: Lim Zhi Hao
paper_authors._id: 682d4ccc3b5f51f4218e12b7
paper_authors.name: Eng Siong Chng
paper_title: CS-Sum: A Benchmark for Code-Switching Dialogue Summarization and the
    Limits of Large Language Models
paper_summary: Code-switching (CS) poses a significant challenge for Large Language Models
  (LLMs), yet its comprehensibility remains underexplored in LLMs. We introduce
  CS-Sum, to evaluate the comprehensibility of CS by the LLMs through CS dialogue
  to English summarization. CS-Sum is the first benchmark for CS dialogue
  summarization across Mandarin-English (EN-ZH), Tamil-English (EN-TA), and
  Malay-English (EN-MS), with 900-1300 human-annotated dialogues per language
  pair. Evaluating ten LLMs, including open and closed-source models, we analyze
  performance across few-shot, translate-summarize, and fine-tuning (LoRA, QLoRA
  on synthetic data) approaches. Our findings show that though the scores on
  automated metrics are high, LLMs make subtle mistakes that alter the complete
  meaning of the dialogue. To this end, we introduce 3 most common type of errors
  that LLMs make when handling CS input. Error rates vary across CS pairs and
  LLMs, with some LLMs showing more frequent errors on certain language pairs,
  underscoring the need for specialized training on code-switched data.
paper_ai_summary: LLMs exhibit high automated metric scores but make subtle errors in code-switching dialogue summarization, highlighting the need for specialized training on CS data.
paper_ai_keywords: LLMs code-switching (CS) CS-Sum Mandarin-English Tamil-English Malay-English dialogue summarization few-shot translate-summarize fine-tuning LoRA QLoRA synthetic data