title: Lost in Cultural Translation: Do LLMs Struggle with Math Across Cultural
    Contexts?
summary: Large Language Models (LLMs) have significantly advanced various fields,
  particularly coding, mathematical reasoning, and logical problem solving.
  However, a critical question remains: Do these mathematical reasoning abilities
  persist when LLMs are presented with culturally adapted math problems?
  Specifically, how do LLMs perform when faced with math problems embedded in
  cultural contexts that have no significant representation in main stream
  web-scale AI training data? To explore this, we generated six synthetic
  cultural datasets from GSM8K, a widely used benchmark for assessing LLMs'
  mathematical reasoning skills. While preserving the mathematical logic and
  numerical values of the original GSM8K test set, we modify cultural elements
  such as personal names, food items, place names, etc. These culturally adapted
  datasets provide a more reliable framework for evaluating LLMs' mathematical
  reasoning under shifting cultural contexts. Our findings reveal that LLMs
  struggle with math problems when cultural references change, even though the
  underlying mathematical structure remains constant. Smaller models exhibit
  greater performance drops compared to larger models. Interestingly, our results
  also suggest that cultural familiarity can enhance mathematical reasoning. Even
  models with no explicit mathematical training but exposure to relevant cultural
  contexts sometimes outperform larger, mathematically proficient models on
  culturally embedded math problems. This study highlights the impact of cultural
  context on the mathematical reasoning abilities of LLMs, underscoring the need
  for more diverse and representative training data to improve robustness in
  real-world applications. The benchmark data sets and script for reproducing the
  results are available at
  https://github.com/akarim23131/Lost_in_Cultural_Translation
paper_authors._id: 67e289d7e5bbf1a125feebf3
paper_authors.name: Aabid Karim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-25T12:19:05.289Z
paper_authors._id: 67e28c6b0bab9de7e86d9a66
paper_authors.avatarUrl: /avatars/d2d56dfe464c8aeb087eeb27572af8f3.svg
paper_authors.fullname: Aabid Karim
paper_authors.type: user
paper_authors.user: abedk
paper_authors._id: 67e289d7e5bbf1a125feebf4
paper_authors.name: Abdul Karim
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-25T11:01:05.131Z
paper_authors._id: 63eb20fbd0b894bbc77de9df
paper_authors.avatarUrl: /avatars/3489e391206d461e880c0dd8adbc6abe.svg
paper_authors.fullname: Abdul Karim
paper_authors.type: user
paper_authors.user: Abdul084
paper_authors._id: 67e289d7e5bbf1a125feebf5
paper_authors.name: Bhoomika Lohana
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-29T03:07:58.082Z
paper_authors._id: 67e76366c1d4f169b8ee59d3
paper_authors.avatarUrl: /avatars/025ddd8fe25957a4959453b456c1d50c.svg
paper_authors.fullname: Bhoomika Lohana
paper_authors.type: user
paper_authors.user: Bhoomikalohana
paper_authors._id: 67e289d7e5bbf1a125feebf6
paper_authors.name: Matt Keon
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-25T10:47:52.707Z
paper_authors._id: 66321599027c1f96a39508df
paper_authors.avatarUrl: /avatars/71dd68d7f0bfc0fac4d047b82de5446a.svg
paper_authors.fullname: Koo
paper_authors.type: user
paper_authors.user: 55mv
paper_authors._id: 67e289d7e5bbf1a125feebf7
paper_authors.name: Jaswinder Singh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-25T12:19:02.963Z
paper_authors._id: 63da27dee697e5898cb73d1a
paper_authors.avatarUrl: /avatars/68b2dffba35e425085e33b4a7cb52a08.svg
paper_authors.fullname: Jaswinder Singh
paper_authors.type: user
paper_authors.user: jaswindersingh2
paper_authors._id: 67e289d7e5bbf1a125feebf8
paper_authors.name: Abdul Sattar
paper_title: Lost in Cultural Translation: Do LLMs Struggle with Math Across Cultural
    Contexts?
paper_summary: Large Language Models (LLMs) have significantly advanced various fields,
  particularly coding, mathematical reasoning, and logical problem solving.
  However, a critical question remains: Do these mathematical reasoning abilities
  persist when LLMs are presented with culturally adapted math problems?
  Specifically, how do LLMs perform when faced with math problems embedded in
  cultural contexts that have no significant representation in main stream
  web-scale AI training data? To explore this, we generated six synthetic
  cultural datasets from GSM8K, a widely used benchmark for assessing LLMs'
  mathematical reasoning skills. While preserving the mathematical logic and
  numerical values of the original GSM8K test set, we modify cultural elements
  such as personal names, food items, place names, etc. These culturally adapted
  datasets provide a more reliable framework for evaluating LLMs' mathematical
  reasoning under shifting cultural contexts. Our findings reveal that LLMs
  struggle with math problems when cultural references change, even though the
  underlying mathematical structure remains constant. Smaller models exhibit
  greater performance drops compared to larger models. Interestingly, our results
  also suggest that cultural familiarity can enhance mathematical reasoning. Even
  models with no explicit mathematical training but exposure to relevant cultural
  contexts sometimes outperform larger, mathematically proficient models on
  culturally embedded math problems. This study highlights the impact of cultural
  context on the mathematical reasoning abilities of LLMs, underscoring the need
  for more diverse and representative training data to improve robustness in
  real-world applications. The benchmark data sets and script for reproducing the
  results are available at
  https://github.com/akarim23131/Lost_in_Cultural_Translation
paper_ai_summary: LLMs encounter challenges with math problems culturally adapted from mainstream AI training data, indicating that cultural context significantly impacts their mathematical reasoning abilities.
paper_ai_keywords: Large Language Models LLMs GSM8K mathematical reasoning cultural context cultural elements cultural adaptation synthetic datasets cultural references cultural familiarity robustness