Audio tokenization model. We use a non-causal five layers EnCodec model for 32 kHz monophonic
audio with a stride of 640, resulting in a frame rate of 50 Hz, and an initial hidden size of 64, doubling
at each of the model’s five layers. The embeddings are quantized with a RVQ with four quantizers,
each with a codebook size of 2048. We follow Défossez et al. [2022] to train the model on one-second
audio segments cropped at random in the audio sequence.
Transformer model. We train autoregressive transformer models at different sizes: 300M, 1.5B,
3.3B parameters. We use a memory efficient Flash attention [Dao et al., 2022] from the xFormers
package [Lefaudeux et al., 2022] to improve both speed and memory usage with long sequences. We
study the impact of the size of the model in Section 4. We use the 300M-parameter model for all of
our ablations. We train on 30-second audio crops sampled at random from the full track. We train
Simple and Controllable Music Generation