title: LINGOLY-TOO: Disentangling Memorisation from Reasoning with Linguistic
    Templatisation and Orthographic Obfuscation
summary: Effective evaluation of the reasoning capabilities of large language models
  (LLMs) are susceptible to overestimation due to data exposure of evaluation
  benchmarks. We introduce a framework for producing linguistic reasoning
  problems that reduces the effect of memorisation in model performance estimates
  and apply this framework to develop LINGOLY-TOO, a challenging evaluation
  benchmark for linguistic reasoning. By developing orthographic templates, we
  dynamically obfuscate the writing systems of real languages to generate
  numerous question variations. These variations preserve the reasoning steps
  required for each solution while reducing the likelihood of specific problem
  instances appearing in model training data. Our experiments demonstrate that
  frontier models, including OpenAI o1-preview and DeepSeem R1, struggle with
  advanced reasoning. Our analysis also shows that LLMs exhibit noticeable
  variance in accuracy across permutations of the same problem, and on average
  perform better on questions appearing in their original orthography. Our
  findings highlight the opaque nature of response generation in LLMs and provide
  evidence that prior data exposure contributes to overestimating the reasoning
  capabilities of frontier models.
paper_authors._id: 67c96a61df4d64bfebd396d4
paper_authors.name: Jude Khouja
paper_authors._id: 67c96a61df4d64bfebd396d5
paper_authors.name: Karolina Korgul
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-07T13:38:07.551Z
paper_authors._id: 64b92ae4ee257c3a4cfbc07a
paper_authors.avatarUrl: /avatars/c4f91978749309c7805e3df1ced115b2.svg
paper_authors.fullname: Karolina Korgul
paper_authors.type: user
paper_authors.user: karotka
paper_authors._id: 67c96a61df4d64bfebd396d6
paper_authors.name: Simi Hellsten
paper_authors._id: 67c96a61df4d64bfebd396d7
paper_authors.name: Lingyi Yang
paper_authors._id: 67c96a61df4d64bfebd396d8
paper_authors.name: Vlad Neacs
paper_authors._id: 67c96a61df4d64bfebd396d9
paper_authors.name: Harry Mayne
paper_authors._id: 67c96a61df4d64bfebd396da
paper_authors.name: Ryan Kearns
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-07T13:38:05.914Z
paper_authors._id: 6751ea855b1cc8f5a14dcc27
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Vw5XZptgrzw2en-xQekAs.png
paper_authors.fullname: Ryan Othniel Kearns
paper_authors.type: user
paper_authors.user: ryanothk
paper_authors._id: 67c96a61df4d64bfebd396db
paper_authors.name: Andrew Bean
paper_authors._id: 67c96a61df4d64bfebd396dc
paper_authors.name: Adam Mahdi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-10T08:03:32.078Z
paper_authors._id: 673cf9c93af47d1d2b6584c9
paper_authors.avatarUrl: /avatars/0792921a8adaa1493255abb0b5aaa4f1.svg
paper_authors.fullname: Adam Mahdi
paper_authors.type: user
paper_authors.user: ammaox
paper_title: LINGOLY-TOO: Disentangling Memorisation from Reasoning with Linguistic
    Templatisation and Orthographic Obfuscation
paper_summary: Effective evaluation of the reasoning capabilities of large language models
  (LLMs) are susceptible to overestimation due to data exposure of evaluation
  benchmarks. We introduce a framework for producing linguistic reasoning
  problems that reduces the effect of memorisation in model performance estimates
  and apply this framework to develop LINGOLY-TOO, a challenging evaluation
  benchmark for linguistic reasoning. By developing orthographic templates, we
  dynamically obfuscate the writing systems of real languages to generate
  numerous question variations. These variations preserve the reasoning steps
  required for each solution while reducing the likelihood of specific problem
  instances appearing in model training data. Our experiments demonstrate that
  frontier models, including OpenAI o1-preview and DeepSeem R1, struggle with
  advanced reasoning. Our analysis also shows that LLMs exhibit noticeable
  variance in accuracy across permutations of the same problem, and on average
  perform better on questions appearing in their original orthography. Our
  findings highlight the opaque nature of response generation in LLMs and provide
  evidence that prior data exposure contributes to overestimating the reasoning
  capabilities of frontier models.
paper_ai_summary: A new evaluation benchmark LINGOLY-TOO is introduced to assess the reasoning capabilities of LLMs by reducing the impact of memorization and prior data exposure.
paper_ai_keywords: large language models LLMs linguistic reasoning evaluation framework LINGOLY-TOO orthographic templates reasoning steps data exposure OpenAI o1-preview DeepSeem R1 response generation