title: Large Scale Transfer Learning for Tabular Data via Language Modeling
summary: Tabular data -- structured, heterogeneous, spreadsheet-style data with rows
  and columns -- is widely used in practice across many domains. However, while
  recent foundation models have reduced the need for developing task-specific
  datasets and predictors in domains such as language modeling and computer
  vision, this transfer learning paradigm has not had similar impact in the
  tabular domain. In this work, we seek to narrow this gap and present TabuLa-8B,
  a language model for tabular prediction. We define a process for extracting a
  large, high-quality training dataset from the TabLib corpus, proposing methods
  for tabular data filtering and quality control. Using the resulting dataset,
  which comprises over 1.6B rows from 3.1M unique tables, we fine-tune a Llama
  3-8B large language model (LLM) for tabular data prediction (classification and
  binned regression) using a novel packing and attention scheme for tabular
  prediction. Through evaluation across a test suite of 329 datasets, we find
  that TabuLa-8B has zero-shot accuracy on unseen tables that is over 15
  percentage points (pp) higher than random guessing, a feat that is not possible
  with existing state-of-the-art tabular prediction models (e.g. XGBoost,
  TabPFN). In the few-shot setting (1-32 shots), without any fine-tuning on the
  target datasets, TabuLa-8B is 5-15 pp more accurate than XGBoost and TabPFN
  models that are explicitly trained on equal, or even up to 16x more data. We
  release our model, code, and data along with the publication of this paper.
paper_authors._id: 66722b866d27e245f6419224
paper_authors.name: Josh Gardner
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-21T07:07:01.674Z
paper_authors._id: 63e6d6b22d2c508de90153a1
paper_authors.avatarUrl: /avatars/11d653e5376bd122ff82778a8393f541.svg
paper_authors.fullname: Josh Gardner
paper_authors.type: user
paper_authors.user: jpgard
paper_authors._id: 66722b866d27e245f6419225
paper_authors.name: Juan C. Perdomo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T21:05:30.311Z
paper_authors._id: 6541496bcb18e4e19ab75765
paper_authors.avatarUrl: /avatars/86311b44583a9a3fcbd3f789132b1fc6.svg
paper_authors.fullname: Juan C Perdomo
paper_authors.type: user
paper_authors.user: jcperdomo
paper_authors._id: 66722b866d27e245f6419226
paper_authors.name: Ludwig Schmidt
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T21:05:50.903Z
paper_authors._id: 6252375c818a5dc29ab9d945
paper_authors.avatarUrl: /avatars/cd4f0b220ea36e13ff71157cffe9d9d1.svg
paper_authors.fullname: Ludwig Schmidt
paper_authors.type: user
paper_authors.user: ludwigschmidt
paper_title: Large Scale Transfer Learning for Tabular Data via Language Modeling
paper_summary: Tabular data -- structured, heterogeneous, spreadsheet-style data with rows
  and columns -- is widely used in practice across many domains. However, while
  recent foundation models have reduced the need for developing task-specific
  datasets and predictors in domains such as language modeling and computer
  vision, this transfer learning paradigm has not had similar impact in the
  tabular domain. In this work, we seek to narrow this gap and present TabuLa-8B,
  a language model for tabular prediction. We define a process for extracting a
  large, high-quality training dataset from the TabLib corpus, proposing methods
  for tabular data filtering and quality control. Using the resulting dataset,
  which comprises over 1.6B rows from 3.1M unique tables, we fine-tune a Llama
  3-8B large language model (LLM) for tabular data prediction (classification and
  binned regression) using a novel packing and attention scheme for tabular
  prediction. Through evaluation across a test suite of 329 datasets, we find
  that TabuLa-8B has zero-shot accuracy on unseen tables that is over 15
  percentage points (pp) higher than random guessing, a feat that is not possible
  with existing state-of-the-art tabular prediction models (e.g. XGBoost,
  TabPFN). In the few-shot setting (1-32 shots), without any fine-tuning on the
  target datasets, TabuLa-8B is 5-15 pp more accurate than XGBoost and TabPFN
  models that are explicitly trained on equal, or even up to 16x more data. We
  release our model, code, and data along with the publication of this paper.
paper_ai_summary: A transformer-based language model, TabuLa-8B, achieves superior zero-shot and few-shot prediction accuracy on tabular data compared to existing models.
paper_ai_keywords: TabuLa-8B TabLib corpus tabular data prediction Llama 3-8B packing and attention scheme zero-shot accuracy few-shot setting XGBoost TabPFN