Then, we immediately have that U and ˜U follow the same distribution. This means that if we can fit a
perfect estimate ˆp of p with a deep learning model, then we can fit exactly the distribution of U.
As stated before, the main issue with the representation Q we obtained from the EnCodec model is
that there are K codebooks for each time step. One solution would be to flatten out Q, thus taking
S = d · fr · K, e.g. first predicting the first codebook of the first time step, then the second codebook
of the first time step, etc. Then, using eq. (1) and eq. (2), we could theoretically fit an exact model
of the distribution of Q. The downside however is the increased complexity, with part of the gain
coming from the lowest sample rate fr being lost.
More than one possible flattening exists, and not all the ˆpt functions need to be estimated through a
single model. For instance, MusicLM [Agostinelli et al., 2023] uses two models, one modeling the
Simple and Controllable Music Generation