Why In-Context Learning Transformers are Tabular Data Classifiers
<|reference_start|>Why In-Context Learning Transformers are Tabular Data Classifiers: The recently introduced TabPFN pretrains an In-Context Learning (ICL) transformer on synthetic data to perform tabular data classification. As synthetic data does not share features or labels with real-world data, the underlying mechanism that contributes to the success of this method remains unclear. This study provides an explanation by demonstrating that ICL-transformers acquire the ability to create complex decision boundaries during pretraining. To validate our claim, we develop a novel forest dataset generator which creates datasets that are unrealistic, but have complex decision boundaries. Our experiments confirm the effectiveness of ICL-transformers pretrained on this data. Furthermore, we create TabForestPFN, the ICL-transformer pretrained on both the original TabPFN synthetic dataset generator and our forest dataset generator. By fine-tuning this model, we reach the current state-of-the-art on tabular data classification. Code is available at https://github.com/FelixdenBreejen/TabForestPFN.<|reference_end|>
@article{breejen2024why,
    title={Why In-Context Learning Transformers are Tabular Data Classifiers},
    author={Felix den Breejen, Sangmin Bae, Stephen Cha, Se-Young Yun},
    journal={arXiv preprint arXiv:2405.13396},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.13396},
    primaryClass={cs.LG stat.ML}
}