title: Discrete Flow Matching
summary: Despite Flow Matching and diffusion models having emerged as powerful
  generative paradigms for continuous variables such as images and videos, their
  application to high-dimensional discrete data, such as language, is still
  limited. In this work, we present Discrete Flow Matching, a novel discrete flow
  paradigm designed specifically for generating discrete data. Discrete Flow
  Matching offers several key contributions: (i) it works with a general family
  of probability paths interpolating between source and target distributions;
  (ii) it allows for a generic formula for sampling from these probability paths
  using learned posteriors such as the probability denoiser (x-prediction) and
  noise-prediction (epsilon-prediction); (iii) practically, focusing on
  specific probability paths defined with different schedulers considerably
  improves generative perplexity compared to previous discrete diffusion and flow
  models; and (iv) by scaling Discrete Flow Matching models up to 1.7B
  parameters, we reach 6.7% Pass@1 and 13.4% Pass@10 on HumanEval and 6.7% Pass@1
  and 20.6% Pass@10 on 1-shot MBPP coding benchmarks. Our approach is capable of
  generating high-quality discrete data in a non-autoregressive fashion,
  significantly closing the gap between autoregressive models and discrete flow
  models.
paper_authors._id: 669fa3551d9ef5e557184206
paper_authors.name: Itai Gat
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T12:46:20.616Z
paper_authors._id: 62a8fa984d933c74bf410c16
paper_authors.avatarUrl: /avatars/73519deba3176be9c23d49f749aee5da.svg
paper_authors.fullname: Itai Gat
paper_authors.type: user
paper_authors.user: itaigat
paper_authors._id: 669fa3551d9ef5e557184207
paper_authors.name: Tal Remez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T12:46:26.570Z
paper_authors._id: 647f1b66838ac3601fc762d1
paper_authors.avatarUrl: /avatars/24aaaeb700690bc84ad0212ce4ae9bd4.svg
paper_authors.fullname: Tal Remez
paper_authors.type: user
paper_authors.user: TalRemez
paper_authors._id: 669fa3551d9ef5e557184208
paper_authors.name: Neta Shaul
paper_authors._id: 669fa3551d9ef5e557184209
paper_authors.name: Felix Kreuk
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T12:46:42.545Z
paper_authors._id: 645d0a4cf4f49de5809de159
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/IOWMd17Iwls0dXsY1OWjK.jpeg
paper_authors.fullname: Felix Kreuk
paper_authors.type: user
paper_authors.user: felixkreuk
paper_authors._id: 669fa3551d9ef5e55718420a
paper_authors.name: Ricky T. Q. Chen
paper_authors._id: 669fa3551d9ef5e55718420b
paper_authors.name: Gabriel Synnaeve
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T12:47:07.756Z
paper_authors._id: 630eac7931970d1cd4fbacf2
paper_authors.avatarUrl: /avatars/b7ccbddfa745db854dc342be1327cd53.svg
paper_authors.fullname: Gabriel Synnaeve
paper_authors.type: user
paper_authors.user: gsynnaeve
paper_authors._id: 669fa3551d9ef5e55718420c
paper_authors.name: Yossi Adi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T12:47:13.577Z
paper_authors._id: 6481e135578646b5c2386728
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6481e135578646b5c2386728/SPva4iNw0pORiCXD45cx9.jpeg
paper_authors.fullname: Yossi Adi
paper_authors.type: user
paper_authors.user: adiyoss
paper_authors._id: 669fa3551d9ef5e55718420d
paper_authors.name: Yaron Lipman
paper_title: Discrete Flow Matching
paper_summary: Despite Flow Matching and diffusion models having emerged as powerful
  generative paradigms for continuous variables such as images and videos, their
  application to high-dimensional discrete data, such as language, is still
  limited. In this work, we present Discrete Flow Matching, a novel discrete flow
  paradigm designed specifically for generating discrete data. Discrete Flow
  Matching offers several key contributions: (i) it works with a general family
  of probability paths interpolating between source and target distributions;
  (ii) it allows for a generic formula for sampling from these probability paths
  using learned posteriors such as the probability denoiser (x-prediction) and
  noise-prediction (epsilon-prediction); (iii) practically, focusing on
  specific probability paths defined with different schedulers considerably
  improves generative perplexity compared to previous discrete diffusion and flow
  models; and (iv) by scaling Discrete Flow Matching models up to 1.7B
  parameters, we reach 6.7% Pass@1 and 13.4% Pass@10 on HumanEval and 6.7% Pass@1
  and 20.6% Pass@10 on 1-shot MBPP coding benchmarks. Our approach is capable of
  generating high-quality discrete data in a non-autoregressive fashion,
  significantly closing the gap between autoregressive models and discrete flow
  models.
paper_ai_keywords: Flow Matching diffusion models generative paradigms continuous variables high-dimensional discrete data Discrete Flow Matching discrete flow paradigm probability paths source and target distributions sampling learned posteriors probability denoiser noise-prediction generative perplexity Discrete Flow Matching models HumanEval MBPP coding benchmarks non-autoregressive fashion