title: Farzi Data: Autoregressive Data Distillation
summary: We study data distillation for auto-regressive machine learning tasks, where
  the input and output have a strict left-to-right causal structure. More
  specifically, we propose Farzi, which summarizes an event sequence dataset into
  a small number of synthetic sequences -- Farzi Data -- which are optimized to
  maintain (if not improve) model performance compared to training on the full
  dataset. Under the hood, Farzi conducts memory-efficient data distillation by
  (i) deriving efficient reverse-mode differentiation of the Adam optimizer by
  leveraging Hessian-Vector Products; and (ii) factorizing the high-dimensional
  discrete event-space into a latent-space which provably promotes implicit
  regularization. Empirically, for sequential recommendation and language
  modeling tasks, we are able to achieve 98-120% of downstream full-data
  performance when training state-of-the-art models on Farzi Data of size as
  little as 0.1% of the original dataset. Notably, being able to train better
  models with significantly less data sheds light on the design of future large
  auto-regressive models, and opens up new opportunities to further scale up
  model and data sizes.
paper_authors._id: 652deefa2d617ec278bfee1d
paper_authors.name: Noveen Sachdeva
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-17T15:11:52.497Z
paper_authors._id: 652e529acb01191873067b02
paper_authors.avatarUrl: /avatars/a340d5faca6ad1d120e8a904380276e9.svg
paper_authors.fullname: Sachdeva
paper_authors.type: user
paper_authors.user: Noveen
paper_authors._id: 652deefa2d617ec278bfee1e
paper_authors.name: Zexue He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T08:05:14.295Z
paper_authors._id: 64d3f0462ad1d0608d95550f
paper_authors.avatarUrl: /avatars/75055e03ceade47a50f5111e27079a3a.svg
paper_authors.fullname: Zexue He
paper_authors.type: user
paper_authors.user: JJesssie
paper_authors._id: 652deefa2d617ec278bfee1f
paper_authors.name: Wang-Cheng Kang
paper_authors._id: 652deefa2d617ec278bfee20
paper_authors.name: Jianmo Ni
paper_authors._id: 652deefa2d617ec278bfee21
paper_authors.name: Derek Zhiyuan Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T08:06:11.115Z
paper_authors._id: 624cedaa43ce7877c5af2c4e
paper_authors.avatarUrl: /avatars/86f13f8e082e5c0b8ea671c374e4d675.svg
paper_authors.fullname: Zhiyuan Cheng
paper_authors.type: user
paper_authors.user: willcee
paper_authors._id: 652deefa2d617ec278bfee22
paper_authors.name: Julian McAuley
paper_title: Farzi Data: Autoregressive Data Distillation
paper_summary: We study data distillation for auto-regressive machine learning tasks, where
  the input and output have a strict left-to-right causal structure. More
  specifically, we propose Farzi, which summarizes an event sequence dataset into
  a small number of synthetic sequences -- Farzi Data -- which are optimized to
  maintain (if not improve) model performance compared to training on the full
  dataset. Under the hood, Farzi conducts memory-efficient data distillation by
  (i) deriving efficient reverse-mode differentiation of the Adam optimizer by
  leveraging Hessian-Vector Products; and (ii) factorizing the high-dimensional
  discrete event-space into a latent-space which provably promotes implicit
  regularization. Empirically, for sequential recommendation and language
  modeling tasks, we are able to achieve 98-120% of downstream full-data
  performance when training state-of-the-art models on Farzi Data of size as
  little as 0.1% of the original dataset. Notably, being able to train better
  models with significantly less data sheds light on the design of future large
  auto-regressive models, and opens up new opportunities to further scale up
  model and data sizes.
paper_ai_summary: Farzi distills large event sequence datasets into smaller synthetic datasets, significantly reducing data size while maintaining high model performance for auto-regressive tasks.
paper_ai_keywords: Farzi data distillation auto-regressive tasks event sequence synthetic sequences Adam optimizer reverse-mode differentiation Hessian-Vector Products latent-space implicit regularization sequential recommendation language modeling