Masked Audio Generation using a Single Non-Autoregressive Transformer
We introduce MAGNeT, a masked generative sequence modeling method that
operates directly over several streams of audio tokens. Unlike prior work,
MAGNeT is comprised of a single-stage, non-autoregressive transformer. During
training, we predict spans of masked tokens obtained from a masking scheduler,
while during inference we gradually construct the output sequence using several
decoding steps. To further enhance the quality of the generated audio, we
introduce a novel rescoring method in which, we leverage an external
pre-trained model to rescore and rank predictions from MAGNeT, which will be
then used for later decoding steps. Lastly, we explore a hybrid version of
MAGNeT, in which we fuse between autoregressive and non-autoregressive models
to generate the first few seconds in an autoregressive manner while the rest of
the sequence is being decoded in parallel. We demonstrate the efficiency of
MAGNeT for the task of text-to-music and text-to-audio generation and conduct
an extensive empirical evaluation, considering both objective metrics and human
studies. The proposed approach is comparable to the evaluated baselines, while
being significantly faster (x7 faster than the autoregressive baseline).
Through ablation studies and analysis, we shed light on the importance of each
of the components comprising MAGNeT, together with pointing to the trade-offs
between autoregressive and non-autoregressive modeling, considering latency,
throughput, and generation quality. Samples are available on our demo page
https://pages.cs.huji.ac.il/adiyoss-lab/MAGNeT.
659e0f2b1692b39ff0bbfa72
Alon Ziv
admin_assigned
2024-01-10T09:04:48.360Z
659d5f32a541d91a57ce7ae0
/avatars/120ac6417b73627e98488afdc715227b.svg
Alon Ziv
user
alonzi
659e0f2b1692b39ff0bbfa73
Itai Gat
admin_assigned
2024-01-10T09:05:02.069Z
62a8fa984d933c74bf410c16
/avatars/73519deba3176be9c23d49f749aee5da.svg
Itai Gat
user
itaigat
659e0f2b1692b39ff0bbfa74
Gael Le Lan
659e0f2b1692b39ff0bbfa75
Tal Remez
admin_assigned
2024-01-10T09:05:29.272Z
647f1b66838ac3601fc762d1
/avatars/24aaaeb700690bc84ad0212ce4ae9bd4.svg
Tal Remez
user
TalRemez
659e0f2b1692b39ff0bbfa76
Felix Kreuk
admin_assigned
2024-01-10T09:05:35.720Z
645d0a4cf4f49de5809de159
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/IOWMd17Iwls0dXsY1OWjK.jpeg
Felix Kreuk
user
felixkreuk
659e0f2b1692b39ff0bbfa77
Alexandre Défossez
admin_assigned
2024-01-10T09:05:41.700Z
6357f5f50dce5559211a3503
https://cdn-avatars.huggingface.co/v1/production/uploads/1666708948380-noauth.jpeg
Alexandre Défossez
user
adefossez
659e0f2b1692b39ff0bbfa78
Jade Copet
admin_assigned
2024-01-10T09:05:47.556Z
6481e0ac50b759c75d5fdad0
/avatars/49f08d989ca505ae01bce5578a94f6fe.svg
Jade Copet
user
JadeCopet
659e0f2b1692b39ff0bbfa79
Gabriel Synnaeve
admin_assigned
2024-01-10T09:05:54.092Z
630eac7931970d1cd4fbacf2
/avatars/b7ccbddfa745db854dc342be1327cd53.svg
Gabriel Synnaeve
user
gsynnaeve
659e0f2b1692b39ff0bbfa7a
Yossi Adi
admin_assigned
2024-01-10T09:06:00.172Z
6481e135578646b5c2386728
https://cdn-avatars.huggingface.co/v1/production/uploads/6481e135578646b5c2386728/SPva4iNw0pORiCXD45cx9.jpeg
Yossi Adi
user
adiyoss
Masked Audio Generation using a Single Non-Autoregressive Transformer
We introduce MAGNeT, a masked generative sequence modeling method that
operates directly over several streams of audio tokens. Unlike prior work,
MAGNeT is comprised of a single-stage, non-autoregressive transformer. During
training, we predict spans of masked tokens obtained from a masking scheduler,
while during inference we gradually construct the output sequence using several
decoding steps. To further enhance the quality of the generated audio, we
introduce a novel rescoring method in which, we leverage an external
pre-trained model to rescore and rank predictions from MAGNeT, which will be
then used for later decoding steps. Lastly, we explore a hybrid version of
MAGNeT, in which we fuse between autoregressive and non-autoregressive models
to generate the first few seconds in an autoregressive manner while the rest of
the sequence is being decoded in parallel. We demonstrate the efficiency of
MAGNeT for the task of text-to-music and text-to-audio generation and conduct
an extensive empirical evaluation, considering both objective metrics and human
studies. The proposed approach is comparable to the evaluated baselines, while
being significantly faster (x7 faster than the autoregressive baseline).
Through ablation studies and analysis, we shed light on the importance of each
of the components comprising MAGNeT, together with pointing to the trade-offs
between autoregressive and non-autoregressive modeling, considering latency,
throughput, and generation quality. Samples are available on our demo page
https://pages.cs.huji.ac.il/adiyoss-lab/MAGNeT.
MAGNeT is a single-stage, non-autoregressive transformer for audio generation that uses masked spans during training, a rescoring method, and a hybrid approach to achieve fast generation.
masked generative sequence modeling non-autoregressive transformer masking scheduler decoding steps rescoring method external pre-trained model hybrid version autoregressive model text-to-music text-to-audio generation ablation studies