nowadays, a more natural approach for music is conditioning on a melodic structure from another
audio track or even whistling or humming. Such an approach also allows for an iterative refinement
of the model’s output. To support that, we experiment with controlling the melodic structure via
jointly conditioning on the input’s chromagram and text description. In preliminary experiments, we
observed that conditioning on the raw chromagram often led to reconstructing the original sample,
resulting in overfitting. To reduce it, we introduce an information bottleneck by choosing the dominant
time-frequency bin in each time step. While a similar capability was shown in Agostinelli et al.
[2023], the authors used supervised proprietary data, which is tedious and costly to collect. In this
work, we take an unsupervised approach, eliminating the requirement for supervised data.
Simple and Controllable Music Generation