On Memorization of Large Language Models in Logical Reasoning
Large language models (LLMs) achieve good performance on challenging
reasoning benchmarks, yet could also make basic reasoning mistakes. This
contrasting behavior is puzzling when it comes to understanding the mechanisms
behind LLMs' reasoning capabilities. One hypothesis is that the increasingly
high and nearly saturated performance on common reasoning benchmarks could be
due to the memorization of similar problems. In this paper, we systematically
investigate this hypothesis with a quantitative measurement of memorization in
reasoning tasks, using a dynamically generated logical reasoning benchmark
based on Knights and Knaves (K&K) puzzles. We found that LLMs could interpolate
the training puzzles (achieving near-perfect accuracy) after fine-tuning, yet
fail when those puzzles are slightly perturbed, suggesting that the models
heavily rely on memorization to solve those training puzzles. On the other
hand, we show that while fine-tuning leads to heavy memorization, it also
consistently improves generalization performance. In-depth analyses with
perturbation tests, cross difficulty-level transferability, probing model
internals, and fine-tuning with wrong answers suggest that the LLMs learn to
reason on K&K puzzles despite training data memorization. This phenomenon
indicates that LLMs exhibit a complex interplay between memorization and
genuine reasoning abilities. Finally, our analysis with per-sample memorization
score sheds light on how LLMs switch between reasoning and memorization in
solving logical puzzles. Our code and data are available at
https://memkklogic.github.io.
6723caf635c2e645d73c8f98
Chulin Xie
6723caf635c2e645d73c8f99
Yangsibo Huang
claimed_verified
2024-11-03T14:22:42.217Z
645949e7ecf89b6a375ff129
/avatars/93f167fe70c43328b95ed597b6dfa51b.svg
Yangsibo Huang
user
yangsibo
6723caf635c2e645d73c8f9a
Chiyuan Zhang
6723caf635c2e645d73c8f9b
Da Yu
6723caf635c2e645d73c8f9c
Xinyun Chen
6723caf635c2e645d73c8f9d
Bill Yuchen Lin
claimed_verified
2024-11-03T14:22:40.499Z
607f666a4ad99100d63ce35c
https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
Bill Yuchen Lin
user
yuchenlin
6723caf635c2e645d73c8f9e
Bo Li
6723caf635c2e645d73c8f9f
Badih Ghazi
6723caf635c2e645d73c8fa0
Ravi Kumar
On Memorization of Large Language Models in Logical Reasoning
Large language models (LLMs) achieve good performance on challenging
reasoning benchmarks, yet could also make basic reasoning mistakes. This
contrasting behavior is puzzling when it comes to understanding the mechanisms
behind LLMs' reasoning capabilities. One hypothesis is that the increasingly
high and nearly saturated performance on common reasoning benchmarks could be
due to the memorization of similar problems. In this paper, we systematically
investigate this hypothesis with a quantitative measurement of memorization in
reasoning tasks, using a dynamically generated logical reasoning benchmark
based on Knights and Knaves (K&K) puzzles. We found that LLMs could interpolate
the training puzzles (achieving near-perfect accuracy) after fine-tuning, yet
fail when those puzzles are slightly perturbed, suggesting that the models
heavily rely on memorization to solve those training puzzles. On the other
hand, we show that while fine-tuning leads to heavy memorization, it also
consistently improves generalization performance. In-depth analyses with
perturbation tests, cross difficulty-level transferability, probing model
internals, and fine-tuning with wrong answers suggest that the LLMs learn to
reason on K&K puzzles despite training data memorization. This phenomenon
indicates that LLMs exhibit a complex interplay between memorization and
genuine reasoning abilities. Finally, our analysis with per-sample memorization
score sheds light on how LLMs switch between reasoning and memorization in
solving logical puzzles. Our code and data are available at
https://memkklogic.github.io.
LLMs demonstrate both near-perfect memorization of training puzzles and improved generalization after fine-tuning, indicating a complex interplay between memorization and genuine reasoning.
LLMs memorization reasoning benchmarks fine-tuning DynaK&K perturbation tests cross difficulty-level transferability probing model internals per-sample memorization score