title: Characterizing Bias: Benchmarking Large Language Models in Simplified
    versus Traditional Chinese
summary: While the capabilities of Large Language Models (LLMs) have been studied in
  both Simplified and Traditional Chinese, it is yet unclear whether LLMs exhibit
  differential performance when prompted in these two variants of written
  Chinese. This understanding is critical, as disparities in the quality of LLM
  responses can perpetuate representational harms by ignoring the different
  cultural contexts underlying Simplified versus Traditional Chinese, and can
  exacerbate downstream harms in LLM-facilitated decision-making in domains such
  as education or hiring. To investigate potential LLM performance disparities,
  we design two benchmark tasks that reflect real-world scenarios: regional term
  choice (prompting the LLM to name a described item which is referred to
  differently in Mainland China and Taiwan), and regional name choice (prompting
  the LLM to choose who to hire from a list of names in both Simplified and
  Traditional Chinese). For both tasks, we audit the performance of 11 leading
  commercial LLM services and open-sourced models -- spanning those primarily
  trained on English, Simplified Chinese, or Traditional Chinese. Our analyses
  indicate that biases in LLM responses are dependent on both the task and
  prompting language: while most LLMs disproportionately favored Simplified
  Chinese responses in the regional term choice task, they surprisingly favored
  Traditional Chinese names in the regional name choice task. We find that these
  disparities may arise from differences in training data representation, written
  character preferences, and tokenization of Simplified and Traditional Chinese.
  These findings highlight the need for further analysis of LLM biases; as such,
  we provide an open-sourced benchmark dataset to foster reproducible evaluations
  of future LLM behavior across Chinese language variants
  (https://github.com/brucelyu17/SC-TC-Bench).
paper_authors._id: 6837dbed1233747046da00f5
paper_authors.name: Hanjia Lyu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T20:19:00.877Z
paper_authors._id: 64c939307dba66c3a7e4d215
paper_authors.avatarUrl: /avatars/0b662cc1799525188476f3e6e1f97d29.svg
paper_authors.fullname: BruceLyu
paper_authors.type: user
paper_authors.user: brucelyu
paper_authors._id: 6837dbed1233747046da00f6
paper_authors.name: Jiebo Luo
paper_authors._id: 6837dbed1233747046da00f7
paper_authors.name: Jian Kang
paper_authors._id: 6837dbed1233747046da00f8
paper_authors.name: Allison Koenecke
paper_title: Characterizing Bias: Benchmarking Large Language Models in Simplified
    versus Traditional Chinese
paper_summary: While the capabilities of Large Language Models (LLMs) have been studied in
  both Simplified and Traditional Chinese, it is yet unclear whether LLMs exhibit
  differential performance when prompted in these two variants of written
  Chinese. This understanding is critical, as disparities in the quality of LLM
  responses can perpetuate representational harms by ignoring the different
  cultural contexts underlying Simplified versus Traditional Chinese, and can
  exacerbate downstream harms in LLM-facilitated decision-making in domains such
  as education or hiring. To investigate potential LLM performance disparities,
  we design two benchmark tasks that reflect real-world scenarios: regional term
  choice (prompting the LLM to name a described item which is referred to
  differently in Mainland China and Taiwan), and regional name choice (prompting
  the LLM to choose who to hire from a list of names in both Simplified and
  Traditional Chinese). For both tasks, we audit the performance of 11 leading
  commercial LLM services and open-sourced models -- spanning those primarily
  trained on English, Simplified Chinese, or Traditional Chinese. Our analyses
  indicate that biases in LLM responses are dependent on both the task and
  prompting language: while most LLMs disproportionately favored Simplified
  Chinese responses in the regional term choice task, they surprisingly favored
  Traditional Chinese names in the regional name choice task. We find that these
  disparities may arise from differences in training data representation, written
  character preferences, and tokenization of Simplified and Traditional Chinese.
  These findings highlight the need for further analysis of LLM biases; as such,
  we provide an open-sourced benchmark dataset to foster reproducible evaluations
  of future LLM behavior across Chinese language variants
  (https://github.com/brucelyu17/SC-TC-Bench).
paper_ai_summary: Research examines LLM performance biases between Simplified and Traditional Chinese in regional term and name choice tasks, attributing differences to training data and tokenization.
paper_ai_keywords: Large Language Models LLMs LLM-facilitated decision-making regional term choice regional name choice open-sourced benchmark dataset SC-TC-Bench