title: Masked Audio Generation using a Single Non-Autoregressive Transformer
summary: We introduce MAGNeT, a masked generative sequence modeling method that
  operates directly over several streams of audio tokens. Unlike prior work,
  MAGNeT is comprised of a single-stage, non-autoregressive transformer. During
  training, we predict spans of masked tokens obtained from a masking scheduler,
  while during inference we gradually construct the output sequence using several
  decoding steps. To further enhance the quality of the generated audio, we
  introduce a novel rescoring method in which, we leverage an external
  pre-trained model to rescore and rank predictions from MAGNeT, which will be
  then used for later decoding steps. Lastly, we explore a hybrid version of
  MAGNeT, in which we fuse between autoregressive and non-autoregressive models
  to generate the first few seconds in an autoregressive manner while the rest of
  the sequence is being decoded in parallel. We demonstrate the efficiency of
  MAGNeT for the task of text-to-music and text-to-audio generation and conduct
  an extensive empirical evaluation, considering both objective metrics and human
  studies. The proposed approach is comparable to the evaluated baselines, while
  being significantly faster (x7 faster than the autoregressive baseline).
  Through ablation studies and analysis, we shed light on the importance of each
  of the components comprising MAGNeT, together with pointing to the trade-offs
  between autoregressive and non-autoregressive modeling, considering latency,
  throughput, and generation quality. Samples are available on our demo page
  https://pages.cs.huji.ac.il/adiyoss-lab/MAGNeT.
paper_authors._id: 659e0f2b1692b39ff0bbfa72
paper_authors.name: Alon Ziv
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-10T09:04:48.360Z
paper_authors._id: 659d5f32a541d91a57ce7ae0
paper_authors.avatarUrl: /avatars/120ac6417b73627e98488afdc715227b.svg
paper_authors.fullname: Alon Ziv
paper_authors.type: user
paper_authors.user: alonzi
paper_authors._id: 659e0f2b1692b39ff0bbfa73
paper_authors.name: Itai Gat
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-10T09:05:02.069Z
paper_authors._id: 62a8fa984d933c74bf410c16
paper_authors.avatarUrl: /avatars/73519deba3176be9c23d49f749aee5da.svg
paper_authors.fullname: Itai Gat
paper_authors.type: user
paper_authors.user: itaigat
paper_authors._id: 659e0f2b1692b39ff0bbfa74
paper_authors.name: Gael Le Lan
paper_authors._id: 659e0f2b1692b39ff0bbfa75
paper_authors.name: Tal Remez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-10T09:05:29.272Z
paper_authors._id: 647f1b66838ac3601fc762d1
paper_authors.avatarUrl: /avatars/24aaaeb700690bc84ad0212ce4ae9bd4.svg
paper_authors.fullname: Tal Remez
paper_authors.type: user
paper_authors.user: TalRemez
paper_authors._id: 659e0f2b1692b39ff0bbfa76
paper_authors.name: Felix Kreuk
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-10T09:05:35.720Z
paper_authors._id: 645d0a4cf4f49de5809de159
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/IOWMd17Iwls0dXsY1OWjK.jpeg
paper_authors.fullname: Felix Kreuk
paper_authors.type: user
paper_authors.user: felixkreuk
paper_authors._id: 659e0f2b1692b39ff0bbfa77
paper_authors.name: Alexandre Défossez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-10T09:05:41.700Z
paper_authors._id: 6357f5f50dce5559211a3503
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666708948380-noauth.jpeg
paper_authors.fullname: Alexandre Défossez
paper_authors.type: user
paper_authors.user: adefossez
paper_authors._id: 659e0f2b1692b39ff0bbfa78
paper_authors.name: Jade Copet
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-10T09:05:47.556Z
paper_authors._id: 6481e0ac50b759c75d5fdad0
paper_authors.avatarUrl: /avatars/49f08d989ca505ae01bce5578a94f6fe.svg
paper_authors.fullname: Jade Copet
paper_authors.type: user
paper_authors.user: JadeCopet
paper_authors._id: 659e0f2b1692b39ff0bbfa79
paper_authors.name: Gabriel Synnaeve
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-10T09:05:54.092Z
paper_authors._id: 630eac7931970d1cd4fbacf2
paper_authors.avatarUrl: /avatars/b7ccbddfa745db854dc342be1327cd53.svg
paper_authors.fullname: Gabriel Synnaeve
paper_authors.type: user
paper_authors.user: gsynnaeve
paper_authors._id: 659e0f2b1692b39ff0bbfa7a
paper_authors.name: Yossi Adi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-10T09:06:00.172Z
paper_authors._id: 6481e135578646b5c2386728
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6481e135578646b5c2386728/SPva4iNw0pORiCXD45cx9.jpeg
paper_authors.fullname: Yossi Adi
paper_authors.type: user
paper_authors.user: adiyoss
paper_title: Masked Audio Generation using a Single Non-Autoregressive Transformer
paper_summary: We introduce MAGNeT, a masked generative sequence modeling method that
  operates directly over several streams of audio tokens. Unlike prior work,
  MAGNeT is comprised of a single-stage, non-autoregressive transformer. During
  training, we predict spans of masked tokens obtained from a masking scheduler,
  while during inference we gradually construct the output sequence using several
  decoding steps. To further enhance the quality of the generated audio, we
  introduce a novel rescoring method in which, we leverage an external
  pre-trained model to rescore and rank predictions from MAGNeT, which will be
  then used for later decoding steps. Lastly, we explore a hybrid version of
  MAGNeT, in which we fuse between autoregressive and non-autoregressive models
  to generate the first few seconds in an autoregressive manner while the rest of
  the sequence is being decoded in parallel. We demonstrate the efficiency of
  MAGNeT for the task of text-to-music and text-to-audio generation and conduct
  an extensive empirical evaluation, considering both objective metrics and human
  studies. The proposed approach is comparable to the evaluated baselines, while
  being significantly faster (x7 faster than the autoregressive baseline).
  Through ablation studies and analysis, we shed light on the importance of each
  of the components comprising MAGNeT, together with pointing to the trade-offs
  between autoregressive and non-autoregressive modeling, considering latency,
  throughput, and generation quality. Samples are available on our demo page
  https://pages.cs.huji.ac.il/adiyoss-lab/MAGNeT.
paper_ai_keywords: masked generative sequence modeling single-stage, non-autoregressive transformer spans of masked tokens masking scheduler Decoding steps rescoring method external pre-trained model hybrid MAGNeT autoregressive and non-autoregressive models text-to-music generation text-to-audio generation objective metrics human studies ablation studies generation quality