∗Yossi Adi is Affiliated with both The Hebrew University of Jerusalem & MetaAI.

37th Conference on Neural Information Processing Systems (NeurIPS 2023).

Figure 1: Codebook interleaving patterns presented in Section 2.2. Each time step t1, t2, . . . , tn is
composed of 4 quantized values (corresponding to k1, . . . , k4). When doing autoregressive modelling,
we can flatten or interleave them in various ways, resulting in a new sequence with 4 parallel streams
and steps s1, s2, . . . , sm. The total number of sequence steps S depends on the pattern and original
number of steps T . 0 is a special token indicating empty positions in the pattern.
Simple and Controllable Music Generation