LINGOLY-TOO: Disentangling Memorisation from Reasoning with Linguistic
  Templatisation and Orthographic Obfuscation
Effective evaluation of the reasoning capabilities of large language models
(LLMs) are susceptible to overestimation due to data exposure of evaluation
benchmarks. We introduce a framework for producing linguistic reasoning
problems that reduces the effect of memorisation in model performance estimates
and apply this framework to develop LINGOLY-TOO, a challenging evaluation
benchmark for linguistic reasoning. By developing orthographic templates, we
dynamically obfuscate the writing systems of real languages to generate
numerous question variations. These variations preserve the reasoning steps
required for each solution while reducing the likelihood of specific problem
instances appearing in model training data. Our experiments demonstrate that
frontier models, including OpenAI o1-preview and DeepSeem R1, struggle with
advanced reasoning. Our analysis also shows that LLMs exhibit noticeable
variance in accuracy across permutations of the same problem, and on average
perform better on questions appearing in their original orthography. Our
findings highlight the opaque nature of response generation in LLMs and provide
evidence that prior data exposure contributes to overestimating the reasoning
capabilities of frontier models.
67c96a61df4d64bfebd396d4
Jude Khouja
67c96a61df4d64bfebd396d5
Karolina Korgul
claimed_verified
2025-03-07T13:38:07.551Z
64b92ae4ee257c3a4cfbc07a
/avatars/c4f91978749309c7805e3df1ced115b2.svg
Karolina Korgul
user
karotka
67c96a61df4d64bfebd396d6
Simi Hellsten
67c96a61df4d64bfebd396d7
Lingyi Yang
67c96a61df4d64bfebd396d8
Vlad Neacs
67c96a61df4d64bfebd396d9
Harry Mayne
67c96a61df4d64bfebd396da
Ryan Kearns
claimed_verified
2025-03-07T13:38:05.914Z
6751ea855b1cc8f5a14dcc27
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Vw5XZptgrzw2en-xQekAs.png
Ryan Othniel Kearns
user
ryanothk
67c96a61df4d64bfebd396db
Andrew Bean
67c96a61df4d64bfebd396dc
Adam Mahdi
claimed_verified
2025-03-10T08:03:32.078Z
673cf9c93af47d1d2b6584c9
/avatars/0792921a8adaa1493255abb0b5aaa4f1.svg
Adam Mahdi
user
ammaox
LINGOLY-TOO: Disentangling Memorisation from Reasoning with Linguistic
  Templatisation and Orthographic Obfuscation
Effective evaluation of the reasoning capabilities of large language models
(LLMs) are susceptible to overestimation due to data exposure of evaluation
benchmarks. We introduce a framework for producing linguistic reasoning
problems that reduces the effect of memorisation in model performance estimates
and apply this framework to develop LINGOLY-TOO, a challenging evaluation
benchmark for linguistic reasoning. By developing orthographic templates, we
dynamically obfuscate the writing systems of real languages to generate
numerous question variations. These variations preserve the reasoning steps
required for each solution while reducing the likelihood of specific problem
instances appearing in model training data. Our experiments demonstrate that
frontier models, including OpenAI o1-preview and DeepSeem R1, struggle with
advanced reasoning. Our analysis also shows that LLMs exhibit noticeable
variance in accuracy across permutations of the same problem, and on average
perform better on questions appearing in their original orthography. Our
findings highlight the opaque nature of response generation in LLMs and provide
evidence that prior data exposure contributes to overestimating the reasoning
capabilities of frontier models.
A new evaluation benchmark LINGOLY-TOO is introduced to assess the reasoning capabilities of LLMs by reducing the impact of memorization and prior data exposure.
large language models LLMs linguistic reasoning evaluation framework LINGOLY-TOO orthographic templates reasoning steps data exposure OpenAI o1-preview DeepSeem R1 response generation