title: Elephants Never Forget: Memorization and Learning of Tabular Data in
    Large Language Models
summary: While many have shown how Large Language Models (LLMs) can be applied to a
  diverse set of tasks, the critical issues of data contamination and
  memorization are often glossed over. In this work, we address this concern for
  tabular data. Specifically, we introduce a variety of different techniques to
  assess whether a language model has seen a tabular dataset during training.
  This investigation reveals that LLMs have memorized many popular tabular
  datasets verbatim. We then compare the few-shot learning performance of LLMs on
  datasets that were seen during training to the performance on datasets released
  after training. We find that LLMs perform better on datasets seen during
  training, indicating that memorization leads to overfitting. At the same time,
  LLMs show non-trivial performance on novel datasets and are surprisingly robust
  to data transformations. We then investigate the in-context statistical
  learning abilities of LLMs. Without fine-tuning, we find them to be limited.
  This suggests that much of the few-shot performance on novel datasets is due to
  the LLM's world knowledge. Overall, our results highlight the importance of
  testing whether an LLM has seen an evaluation dataset during pre-training. We
  make the exposure tests we developed available as the tabmemcheck Python
  package at https://github.com/interpretml/LLM-Tabular-Memorization-Checker
paper_authors._id: 6615f5e024c94fd8a222195e
paper_authors.name: Sebastian Bordt
paper_authors._id: 6615f5e024c94fd8a222195f
paper_authors.name: Harsha Nori
paper_authors._id: 6615f5e024c94fd8a2221960
paper_authors.name: Vanessa Rodrigues
paper_authors._id: 6615f5e024c94fd8a2221961
paper_authors.name: Besmira Nushi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T07:59:30.650Z
paper_authors._id: 638ee2ed848625f9fad11034
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1670309301345-638ee2ed848625f9fad11034.jpeg
paper_authors.fullname: Besmira Nushi
paper_authors.type: user
paper_authors.user: nushib
paper_authors._id: 6615f5e024c94fd8a2221962
paper_authors.name: Rich Caruana
paper_title: Elephants Never Forget: Memorization and Learning of Tabular Data in
    Large Language Models
paper_summary: While many have shown how Large Language Models (LLMs) can be applied to a
  diverse set of tasks, the critical issues of data contamination and
  memorization are often glossed over. In this work, we address this concern for
  tabular data. Specifically, we introduce a variety of different techniques to
  assess whether a language model has seen a tabular dataset during training.
  This investigation reveals that LLMs have memorized many popular tabular
  datasets verbatim. We then compare the few-shot learning performance of LLMs on
  datasets that were seen during training to the performance on datasets released
  after training. We find that LLMs perform better on datasets seen during
  training, indicating that memorization leads to overfitting. At the same time,
  LLMs show non-trivial performance on novel datasets and are surprisingly robust
  to data transformations. We then investigate the in-context statistical
  learning abilities of LLMs. Without fine-tuning, we find them to be limited.
  This suggests that much of the few-shot performance on novel datasets is due to
  the LLM's world knowledge. Overall, our results highlight the importance of
  testing whether an LLM has seen an evaluation dataset during pre-training. We
  make the exposure tests we developed available as the tabmemcheck Python
  package at https://github.com/interpretml/LLM-Tabular-Memorization-Checker
paper_ai_summary: LLMs have memorized popular tabular datasets and perform better on seen datasets, indicating overfitting, while still showing robust performance on novel datasets through world knowledge.
paper_ai_keywords: Large Language Models LLMs few-shot learning data contamination memorization in-context statistical learning