The challenge of realistic music generation: modelling raw audio at scale
Realistic music generation is a challenging task. When building generative models of music that are learnt from data, typically high-level representations such as scores or MIDI are used that abstract away the idiosyncrasies of a particular performance. But these nuances are very important for our perception of musicality and realism, so in this work we embark on modelling music in the raw audio domain. It has been shown that autoregressive models excel at generating raw audio waveforms of speech, but when applied to music, we find them biased towards capturing local signal structure at the expense of modelling long-range correlations. This is problematic because music exhibits structure at many different timescales. In this work, we explore autoregressive discrete autoencoders (ADAs) as a means to enable autoregressive models to capture long-range correlations in waveforms. We find that they allow us to unconditionally generate piano music directly in the raw audio domain, which shows stylistic consistency across tens of seconds.
The authors claim that there is no suitable metric to evaluate the quality of the generated audio, which is plausible, so they listened to the audio and evaluated on their own. The only shortcoming here is that no systematic and blind listening test has been conducted yet. The authors themselves might be biased and thus, the capabilities of the proposed approach cannot be considered as fully proven from a scientific perspective. However, a link to the audio is provided so that the readers can convince themselves from the proposed method.  Minor comments: -"nats per timestep": should be defined -p. 3, l. 115-116: The reviewer is not sure, whether the term "hop size" is suitable to describe the ratio between the sample rates. -p. 7, l. 278: For AMEA models, PBT turns (a comma helps the reader) -References: [12], [22], [43] are using "et al." even though the author list is not much longer. All authors should be named to be consistent.   Quality: The authors address a prevailing research topic and has some novel contributions (see below) to the field of audio and music generation. The topic is going along with the recent advances in deep architectures for audio generation.  Clarity: The paper is well written and the proposed methods are explained carefully. A good description of the state-of-the-art in audio generation and their shortcomings are included. The references are complete.  Originality: To the knowledge of the reviewer, in this contribution, music is modelled directly on the waveform for the first time, modelling also long-term dependencies related to consistent musical expression and style. The conditioning signal is learnt with an autoencoder. Moreover, argmax autoencoder is proposed as an alternative (faster and more robust) for the quantisation step employed in vector quantisation variational autoencoders, avoiding the "codebook collapse" problem.  Significance: A systematic evaluation is missing (see above). However, a link to audio samples is provided in the paper and the proposed approach is well-founded.   Confidence: The reviewer is confident with audio processing and machine learning/deep learning and has many peer-reviewed publications in journals and conference proceedings in this field.