title: On Memorization of Large Language Models in Logical Reasoning
summary: Large language models (LLMs) achieve good performance on challenging
  reasoning benchmarks, yet could also make basic reasoning mistakes. This
  contrasting behavior is puzzling when it comes to understanding the mechanisms
  behind LLMs' reasoning capabilities. One hypothesis is that the increasingly
  high and nearly saturated performance on common reasoning benchmarks could be
  due to the memorization of similar problems. In this paper, we systematically
  investigate this hypothesis with a quantitative measurement of memorization in
  reasoning tasks, using a dynamically generated logical reasoning benchmark
  based on Knights and Knaves (K&K) puzzles. We found that LLMs could interpolate
  the training puzzles (achieving near-perfect accuracy) after fine-tuning, yet
  fail when those puzzles are slightly perturbed, suggesting that the models
  heavily rely on memorization to solve those training puzzles. On the other
  hand, we show that while fine-tuning leads to heavy memorization, it also
  consistently improves generalization performance. In-depth analyses with
  perturbation tests, cross difficulty-level transferability, probing model
  internals, and fine-tuning with wrong answers suggest that the LLMs learn to
  reason on K&K puzzles despite training data memorization. This phenomenon
  indicates that LLMs exhibit a complex interplay between memorization and
  genuine reasoning abilities. Finally, our analysis with per-sample memorization
  score sheds light on how LLMs switch between reasoning and memorization in
  solving logical puzzles. Our code and data are available at
  https://memkklogic.github.io.
paper_authors._id: 6723caf635c2e645d73c8f98
paper_authors.name: Chulin Xie
paper_authors._id: 6723caf635c2e645d73c8f99
paper_authors.name: Yangsibo Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-03T14:22:42.217Z
paper_authors._id: 645949e7ecf89b6a375ff129
paper_authors.avatarUrl: /avatars/93f167fe70c43328b95ed597b6dfa51b.svg
paper_authors.fullname: Yangsibo Huang
paper_authors.type: user
paper_authors.user: yangsibo
paper_authors._id: 6723caf635c2e645d73c8f9a
paper_authors.name: Chiyuan Zhang
paper_authors._id: 6723caf635c2e645d73c8f9b
paper_authors.name: Da Yu
paper_authors._id: 6723caf635c2e645d73c8f9c
paper_authors.name: Xinyun Chen
paper_authors._id: 6723caf635c2e645d73c8f9d
paper_authors.name: Bill Yuchen Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-03T14:22:40.499Z
paper_authors._id: 607f666a4ad99100d63ce35c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
paper_authors.fullname: Bill Yuchen Lin
paper_authors.type: user
paper_authors.user: yuchenlin
paper_authors._id: 6723caf635c2e645d73c8f9e
paper_authors.name: Bo Li
paper_authors._id: 6723caf635c2e645d73c8f9f
paper_authors.name: Badih Ghazi
paper_authors._id: 6723caf635c2e645d73c8fa0
paper_authors.name: Ravi Kumar
paper_title: On Memorization of Large Language Models in Logical Reasoning
paper_summary: Large language models (LLMs) achieve good performance on challenging
  reasoning benchmarks, yet could also make basic reasoning mistakes. This
  contrasting behavior is puzzling when it comes to understanding the mechanisms
  behind LLMs' reasoning capabilities. One hypothesis is that the increasingly
  high and nearly saturated performance on common reasoning benchmarks could be
  due to the memorization of similar problems. In this paper, we systematically
  investigate this hypothesis with a quantitative measurement of memorization in
  reasoning tasks, using a dynamically generated logical reasoning benchmark
  based on Knights and Knaves (K&K) puzzles. We found that LLMs could interpolate
  the training puzzles (achieving near-perfect accuracy) after fine-tuning, yet
  fail when those puzzles are slightly perturbed, suggesting that the models
  heavily rely on memorization to solve those training puzzles. On the other
  hand, we show that while fine-tuning leads to heavy memorization, it also
  consistently improves generalization performance. In-depth analyses with
  perturbation tests, cross difficulty-level transferability, probing model
  internals, and fine-tuning with wrong answers suggest that the LLMs learn to
  reason on K&K puzzles despite training data memorization. This phenomenon
  indicates that LLMs exhibit a complex interplay between memorization and
  genuine reasoning abilities. Finally, our analysis with per-sample memorization
  score sheds light on how LLMs switch between reasoning and memorization in
  solving logical puzzles. Our code and data are available at
  https://memkklogic.github.io.
paper_ai_summary: LLMs demonstrate both near-perfect memorization of training puzzles and improved generalization after fine-tuning, indicating a complex interplay between memorization and genuine reasoning.
paper_ai_keywords: LLMs memorization reasoning benchmarks fine-tuning DynaK&K perturbation tests cross difficulty-level transferability probing model internals per-sample memorization score