title: Tabby: Tabular Data Synthesis with Language Models
summary: While advances in large language models (LLMs) have greatly improved the
  quality of synthetic text data in recent years, synthesizing tabular data has
  received relatively less attention. We address this disparity with Tabby, a
  simple but powerful post-training modification to the standard Transformer
  language model architecture, enabling its use for tabular dataset synthesis.
  Tabby enables the representation of differences across columns using Gated
  Mixture-of-Experts, with column-specific sets of parameters. Empirically, Tabby
  results in data quality near or equal to that of real data. By pairing our
  novel LLM table training technique, Plain, with Tabby, we observe up to a 44%
  improvement in quality over previous methods. We also show that Tabby extends
  beyond tables to more general structured data, reaching parity with real data
  on a nested JSON dataset as well.
paper_authors._id: 67c8691bb43e75ced9747fe6
paper_authors.name: Sonia Cromp
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-05T15:57:21.571Z
paper_authors._id: 63865c918b5acae8d24ffff3
paper_authors.avatarUrl: /avatars/7916f38ce38266a0a3b39a54f75636ff.svg
paper_authors.fullname: Sonia
paper_authors.type: user
paper_authors.user: sonicc
paper_authors._id: 67c8691bb43e75ced9747fe7
paper_authors.name: Satya Sai Srinath Namburi GNVV
paper_authors._id: 67c8691bb43e75ced9747fe8
paper_authors.name: Mohammed Alkhudhayri
paper_authors._id: 67c8691bb43e75ced9747fe9
paper_authors.name: Catherine Cao
paper_authors._id: 67c8691bb43e75ced9747fea
paper_authors.name: Samuel Guo
paper_authors._id: 67c8691bb43e75ced9747feb
paper_authors.name: Nicholas Roberts
paper_authors._id: 67c8691bb43e75ced9747fec
paper_authors.name: Frederic Sala
paper_title: Tabby: Tabular Data Synthesis with Language Models
paper_summary: While advances in large language models (LLMs) have greatly improved the
  quality of synthetic text data in recent years, synthesizing tabular data has
  received relatively less attention. We address this disparity with Tabby, a
  simple but powerful post-training modification to the standard Transformer
  language model architecture, enabling its use for tabular dataset synthesis.
  Tabby enables the representation of differences across columns using Gated
  Mixture-of-Experts, with column-specific sets of parameters. Empirically, Tabby
  results in data quality near or equal to that of real data. By pairing our
  novel LLM table training technique, Plain, with Tabby, we observe up to a 44%
  improvement in quality over previous methods. We also show that Tabby extends
  beyond tables to more general structured data, reaching parity with real data
  on a nested JSON dataset as well.
paper_ai_summary: Tabby, a post-training modification using Gated Mixture-of-Experts, enhances Transformer models for tabular and structured data synthesis, achieving high data quality.
paper_ai_keywords: Transformer Gated Mixture-of-Experts Tabby Plain