Lost in Cultural Translation: Do LLMs Struggle with Math Across Cultural
  Contexts?
Large Language Models (LLMs) have significantly advanced various fields,
particularly coding, mathematical reasoning, and logical problem solving.
However, a critical question remains: Do these mathematical reasoning abilities
persist when LLMs are presented with culturally adapted math problems?
Specifically, how do LLMs perform when faced with math problems embedded in
cultural contexts that have no significant representation in main stream
web-scale AI training data? To explore this, we generated six synthetic
cultural datasets from GSM8K, a widely used benchmark for assessing LLMs'
mathematical reasoning skills. While preserving the mathematical logic and
numerical values of the original GSM8K test set, we modify cultural elements
such as personal names, food items, place names, etc. These culturally adapted
datasets provide a more reliable framework for evaluating LLMs' mathematical
reasoning under shifting cultural contexts. Our findings reveal that LLMs
struggle with math problems when cultural references change, even though the
underlying mathematical structure remains constant. Smaller models exhibit
greater performance drops compared to larger models. Interestingly, our results
also suggest that cultural familiarity can enhance mathematical reasoning. Even
models with no explicit mathematical training but exposure to relevant cultural
contexts sometimes outperform larger, mathematically proficient models on
culturally embedded math problems. This study highlights the impact of cultural
context on the mathematical reasoning abilities of LLMs, underscoring the need
for more diverse and representative training data to improve robustness in
real-world applications. The benchmark data sets and script for reproducing the
results are available at
https://github.com/akarim23131/Lost_in_Cultural_Translation
67e289d7e5bbf1a125feebf3
Aabid Karim
claimed_verified
2025-03-25T12:19:05.289Z
67e28c6b0bab9de7e86d9a66
/avatars/d2d56dfe464c8aeb087eeb27572af8f3.svg
Aabid Karim
user
abedk
67e289d7e5bbf1a125feebf4
Abdul Karim
extracted_pending
2025-03-25T11:01:05.131Z
63eb20fbd0b894bbc77de9df
/avatars/3489e391206d461e880c0dd8adbc6abe.svg
Abdul Karim
user
Abdul084
67e289d7e5bbf1a125feebf5
Bhoomika Lohana
extracted_pending
2025-03-29T03:07:58.082Z
67e76366c1d4f169b8ee59d3
/avatars/025ddd8fe25957a4959453b456c1d50c.svg
Bhoomika Lohana
user
Bhoomikalohana
67e289d7e5bbf1a125feebf6
Matt Keon
extracted_pending
2025-03-25T10:47:52.707Z
66321599027c1f96a39508df
/avatars/71dd68d7f0bfc0fac4d047b82de5446a.svg
Koo
user
55mv
67e289d7e5bbf1a125feebf7
Jaswinder Singh
claimed_verified
2025-03-25T12:19:02.963Z
63da27dee697e5898cb73d1a
/avatars/68b2dffba35e425085e33b4a7cb52a08.svg
Jaswinder Singh
user
jaswindersingh2
67e289d7e5bbf1a125feebf8
Abdul Sattar
Lost in Cultural Translation: Do LLMs Struggle with Math Across Cultural
  Contexts?
Large Language Models (LLMs) have significantly advanced various fields,
particularly coding, mathematical reasoning, and logical problem solving.
However, a critical question remains: Do these mathematical reasoning abilities
persist when LLMs are presented with culturally adapted math problems?
Specifically, how do LLMs perform when faced with math problems embedded in
cultural contexts that have no significant representation in main stream
web-scale AI training data? To explore this, we generated six synthetic
cultural datasets from GSM8K, a widely used benchmark for assessing LLMs'
mathematical reasoning skills. While preserving the mathematical logic and
numerical values of the original GSM8K test set, we modify cultural elements
such as personal names, food items, place names, etc. These culturally adapted
datasets provide a more reliable framework for evaluating LLMs' mathematical
reasoning under shifting cultural contexts. Our findings reveal that LLMs
struggle with math problems when cultural references change, even though the
underlying mathematical structure remains constant. Smaller models exhibit
greater performance drops compared to larger models. Interestingly, our results
also suggest that cultural familiarity can enhance mathematical reasoning. Even
models with no explicit mathematical training but exposure to relevant cultural
contexts sometimes outperform larger, mathematically proficient models on
culturally embedded math problems. This study highlights the impact of cultural
context on the mathematical reasoning abilities of LLMs, underscoring the need
for more diverse and representative training data to improve robustness in
real-world applications. The benchmark data sets and script for reproducing the
results are available at
https://github.com/akarim23131/Lost_in_Cultural_Translation
LLMs encounter challenges with math problems culturally adapted from mainstream AI training data, indicating that cultural context significantly impacts their mathematical reasoning abilities.
Large Language Models LLMs GSM8K mathematical reasoning cultural context cultural elements cultural adaptation synthetic datasets cultural references cultural familiarity robustness