CARTE: Pretraining and Transfer for Tabular Learning
<|reference_start|>CARTE: Pretraining and Transfer for Tabular Learning: Pretrained deep-learning models are the go-to solution for images or text. However, for tabular data the standard is still to train tree-based models. Indeed, transfer learning on tables hits the challenge of data integration: finding correspondences, correspondences in the entries (entity matching) where different words may denote the same entity, correspondences across columns (schema matching), which may come in different orders, names... We propose a neural architecture that does not need such correspondences. As a result, we can pretrain it on background data that has not been matched. The architecture -- CARTE for Context Aware Representation of Table Entries -- uses a graph representation of tabular (or relational) data to process tables with different columns, string embedding of entries and columns names to model an open vocabulary, and a graph-attentional network to contextualize entries with column names and neighboring entries. An extensive benchmark shows that CARTE facilitates learning, outperforming a solid set of baselines including the best tree-based models. CARTE also enables joint learning across tables with unmatched columns, enhancing a small table with bigger ones. CARTE opens the door to large pretrained models for tabular data.<|reference_end|>
@article{kim2024carte:,
    title={CARTE: Pretraining and Transfer for Tabular Learning},
    author={Myung Jun Kim, L'eo Grinsztajn, and Ga"el Varoquaux},
    journal={arXiv preprint arXiv:2402.16785},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.16785},
    primaryClass={cs.LG}
}