title: TabSTAR: A Foundation Tabular Model With Semantically Target-Aware
    Representations
summary: While deep learning has achieved remarkable success across many domains, it
  has historically underperformed on tabular learning tasks, which remain
  dominated by gradient boosting decision trees (GBDTs). However, recent
  advancements are paving the way for Tabular Foundation Models, which can
  leverage real-world knowledge and generalize across diverse datasets,
  particularly when the data contains free-text. Although incorporating language
  model capabilities into tabular tasks has been explored, most existing methods
  utilize static, target-agnostic textual representations, limiting their
  effectiveness. We introduce TabSTAR: a Foundation Tabular Model with
  Semantically Target-Aware Representations. TabSTAR is designed to enable
  transfer learning on tabular data with textual features, with an architecture
  free of dataset-specific parameters. It unfreezes a pretrained text encoder and
  takes as input target tokens, which provide the model with the context needed
  to learn task-specific embeddings. TabSTAR achieves state-of-the-art
  performance for both medium- and large-sized datasets across known benchmarks
  of classification tasks with text features, and its pretraining phase exhibits
  scaling laws in the number of datasets, offering a pathway for further
  performance improvements.
paper_authors._id: 6833f8b419852283c4b3bbd6
paper_authors.name: Alan Arazi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T14:07:38.679Z
paper_authors._id: 60d84af7eac5e05d4594f010
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60d84af7eac5e05d4594f010/KnGxUR7OUOAGg0S67tRaY.png
paper_authors.fullname: Alan Arazi
paper_authors.type: user
paper_authors.user: alana89
paper_authors._id: 6833f8b419852283c4b3bbd7
paper_authors.name: Eilam Shapira
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:09:08.206Z
paper_authors._id: 64802fb6c57f629056c59966
paper_authors.avatarUrl: /avatars/d5ecabaceeba759969855acf512b6649.svg
paper_authors.fullname: Eilam Shapira
paper_authors.type: user
paper_authors.user: EilamSha
paper_authors._id: 6833f8b419852283c4b3bbd8
paper_authors.name: Roi Reichart
paper_title: TabSTAR: A Foundation Tabular Model With Semantically Target-Aware
    Representations
paper_summary: While deep learning has achieved remarkable success across many domains, it
  has historically underperformed on tabular learning tasks, which remain
  dominated by gradient boosting decision trees (GBDTs). However, recent
  advancements are paving the way for Tabular Foundation Models, which can
  leverage real-world knowledge and generalize across diverse datasets,
  particularly when the data contains free-text. Although incorporating language
  model capabilities into tabular tasks has been explored, most existing methods
  utilize static, target-agnostic textual representations, limiting their
  effectiveness. We introduce TabSTAR: a Foundation Tabular Model with
  Semantically Target-Aware Representations. TabSTAR is designed to enable
  transfer learning on tabular data with textual features, with an architecture
  free of dataset-specific parameters. It unfreezes a pretrained text encoder and
  takes as input target tokens, which provide the model with the context needed
  to learn task-specific embeddings. TabSTAR achieves state-of-the-art
  performance for both medium- and large-sized datasets across known benchmarks
  of classification tasks with text features, and its pretraining phase exhibits
  scaling laws in the number of datasets, offering a pathway for further
  performance improvements.
paper_ai_summary: TabSTAR, a tabular foundation model with semantically target-aware representations, achieves state-of-the-art performance in classification tasks with text features through transfer learning without dataset-specific parameters.
paper_ai_keywords: TabSTAR foundation tabular model semantically target-aware representations transfer learning pretrained text encoder target tokens task-specific embeddings scaling laws