Large Scale Transfer Learning for Tabular Data via Language Modeling
Tabular data -- structured, heterogeneous, spreadsheet-style data with rows
and columns -- is widely used in practice across many domains. However, while
recent foundation models have reduced the need for developing task-specific
datasets and predictors in domains such as language modeling and computer
vision, this transfer learning paradigm has not had similar impact in the
tabular domain. In this work, we seek to narrow this gap and present TabuLa-8B,
a language model for tabular prediction. We define a process for extracting a
large, high-quality training dataset from the TabLib corpus, proposing methods
for tabular data filtering and quality control. Using the resulting dataset,
which comprises over 1.6B rows from 3.1M unique tables, we fine-tune a Llama
3-8B large language model (LLM) for tabular data prediction (classification and
binned regression) using a novel packing and attention scheme for tabular
prediction. Through evaluation across a test suite of 329 datasets, we find
that TabuLa-8B has zero-shot accuracy on unseen tables that is over 15
percentage points (pp) higher than random guessing, a feat that is not possible
with existing state-of-the-art tabular prediction models (e.g. XGBoost,
TabPFN). In the few-shot setting (1-32 shots), without any fine-tuning on the
target datasets, TabuLa-8B is 5-15 pp more accurate than XGBoost and TabPFN
models that are explicitly trained on equal, or even up to 16x more data. We
release our model, code, and data along with the publication of this paper.
66722b866d27e245f6419224
Josh Gardner
claimed_verified
2024-06-21T07:07:01.674Z
63e6d6b22d2c508de90153a1
/avatars/11d653e5376bd122ff82778a8393f541.svg
Josh Gardner
user
jpgard
66722b866d27e245f6419225
Juan C. Perdomo
admin_assigned
2024-06-19T21:05:30.311Z
6541496bcb18e4e19ab75765
/avatars/86311b44583a9a3fcbd3f789132b1fc6.svg
Juan C Perdomo
user
jcperdomo
66722b866d27e245f6419226
Ludwig Schmidt
admin_assigned
2024-06-19T21:05:50.903Z
6252375c818a5dc29ab9d945
/avatars/cd4f0b220ea36e13ff71157cffe9d9d1.svg
Ludwig Schmidt
user
ludwigschmidt
Large Scale Transfer Learning for Tabular Data via Language Modeling
Tabular data -- structured, heterogeneous, spreadsheet-style data with rows
and columns -- is widely used in practice across many domains. However, while
recent foundation models have reduced the need for developing task-specific
datasets and predictors in domains such as language modeling and computer
vision, this transfer learning paradigm has not had similar impact in the
tabular domain. In this work, we seek to narrow this gap and present TabuLa-8B,
a language model for tabular prediction. We define a process for extracting a
large, high-quality training dataset from the TabLib corpus, proposing methods
for tabular data filtering and quality control. Using the resulting dataset,
which comprises over 1.6B rows from 3.1M unique tables, we fine-tune a Llama
3-8B large language model (LLM) for tabular data prediction (classification and
binned regression) using a novel packing and attention scheme for tabular
prediction. Through evaluation across a test suite of 329 datasets, we find
that TabuLa-8B has zero-shot accuracy on unseen tables that is over 15
percentage points (pp) higher than random guessing, a feat that is not possible
with existing state-of-the-art tabular prediction models (e.g. XGBoost,
TabPFN). In the few-shot setting (1-32 shots), without any fine-tuning on the
target datasets, TabuLa-8B is 5-15 pp more accurate than XGBoost and TabPFN
models that are explicitly trained on equal, or even up to 16x more data. We
release our model, code, and data along with the publication of this paper.
A transformer-based language model, TabuLa-8B, achieves superior zero-shot and few-shot prediction accuracy on tabular data compared to existing models.
TabuLa-8B TabLib corpus tabular data prediction Llama 3-8B packing and attention scheme zero-shot accuracy few-shot setting XGBoost TabPFN