Learning synthesizers and generating music in the raw audio domain is a challenging task. We investigate the learned representations of convolutional autoregressive generative models. Consequently, we show that mappings between musical notes and the harmonic style (instrument timbre) can be learned based on the raw audio music recording and the musical score (in binary piano roll format). Our proposed architecture, SynthNet uses minimal training data (9 minutes), is substantially better in quality and converges 6 times faster than the baselines. The quality of the generated waveforms (generation accuracy) is sufficiently high that they are almost identical to the ground truth. Therefore, we are able to directly measure generation error during training, based on the RMSE of the Constant-Q transform. Mean opinion scores are also provided. We validate our work using 7 distinct harmonic styles and also provide visualizations and links to all generated audio.
This paper proposes a neural model for synthesizing instrument sounds, using an architecture based on the WaveNet and DeepVoice models. The model generates raw waveforms conditioned on a piano roll representation of aligned MIDI input.
My biggest gripe with this work is that the model is trained entirely on a synthetic dataset generated from a sample-based synthesizer using a sound font. I feel that this defeats the purpose, as it will never work better than just sampling the original sound library. One potential argument in favour would be to save storage space, but the sound font used for the work is only ~140 MB, which is not prohibitive these days (indeed, many neural models require a comparable amount of storage).
It would be much more interesting to train the model on real instrument recordings, because then it could capture all the nuances of the instruments that sample-based synthesizers cannot replicate. As it stands, all the model has to do is reproduce a fixed (and fairly small) set of audio samples. This is arguably a much simpler task, which could also explain why reducing the model size (SynthNet's depthwise convolutions have many fewer parameters than the regular convolutions used in WaveNet and DeepVoice) works so well here.
That said, I think the proposed architectural modifications for raw audio models could be interesting and should be tested for other, more challenging tasks. The proposed RMSE-CQT error measure is potentially quite valuable for music generation research, and its correlation with MOS scores is promising (but this should also be tested on more realistic audio).
The fact that the models were trained to convergence on only 9 minutes of data per instrument is also impressive, despite the limitations of the dataset. The use of dithering to reduce perceptual noise is also interesting and some comparison experiments there would have been interesting, especially to corroborate the claim that it is critical for the learning process.
I think the paper slightly overstates its contributions in terms of providing insight into the representations that are learned in generative convolutional models. The Gram matrix projections showing that the activations of different layers diverge for different input types as we advance through the model is not particularly surprising, and similar plots could probably be made for almost any residual model.
Overall, I feel the work has some fundamental flaws, mostly stemming from the dataset that was used.
Miscellany:
- In the abstract: "is substantially better in quality", compared to what?
- In the introduction, it is mentioned that words in a speech signal cannot overlap, but notes in a musical signal can. I would argue that these are not comparable abstractions though, words themselves are composed of a sequence of phonemes, which are probably a better point of comparison (and phonemes, while they don't tend to overlap, can affect neighbouring phonemes in various ways). That said, I appreciate that this is probably quite subjective.
- Overall, the formulation of paragraph 2 of the introduction is a bit unusual, I think the same things are said in a much better way in Section 3.
- "Conditioning Deep Generative Raw Audio Models for Structured Automatic Music" by Manzelli et al. (2018) also proposes a MIDI-conditional neural audio generation model, trained on real instrument recordings from the MusicNet dataset. I think this is a very relevant reference.
- In the contributions of the paper, it is stated that "the generated audio is practically identical to ground truth as can be seen in Figure 4" but the CQTs in this figure are visibly different.
- I don't think it is fair to directly compare this setup to Engel et al. (2017) and Mor et al. (2018) as is done in the last paragraph of Section 2, as these are simply different tasks (mapping from audio to audio as opposed to generating audio).
- At the start of Section 3.1 it would be good to explicitly mention whether 8-bit mu-law audio is used, to explain why the waveform is 256-valued.
- Why is the conditioning causal? It does not need to be, as the piano roll is fully available in advance of the audio generation. I guess one argument in favour would be to enable real-time generation, but it would still be good to compare causal and non-causal conditioning.
- Since the piano roll representation is binary, does that mean MIDI velocity is not captured in the conditioning signal? It would probably be useful for the model to provide this information, so it can capture the differences in timbre between different velocities.
- The use of a MIDI prediction loss to regularise the conditioning part of the model is interesting, but I would have liked to see a comparison experiment (with/without).
- In Section 4.3, specify the unit, i.e. "Delta < 1 second".
- For the task of recreating synthetic audio samples, the WaveNet models seem to be quite large. As far as I can tell the size hyperparameters were chosen based on the literature, but the inherited parameters were originally optimised for different tasks.
- In Section 4.3 under "global conditioning", the benchmark is said to be between DeepVoice L26 and SynthNet L24, but Table 4 lists DeepVoice L26 and SynthNet L26, which version was actually used?