SING: Symbol-to-Instrument Neural Generator
Recent progress in deep learning for audio synthesis opens the way to models that directly produce the waveform, shifting away from the traditional paradigm of relying on vocoders or MIDI synthesizers for speech or music generation. Despite their successes, current state-of-the-art neural audio synthesizers such as WaveNet and SampleRNN [24,17] suffer from prohibitive training and inference times because they are based on autoregressive models that generate audio samples one at a time at a rate of 16kHz. In this work, we study the more computationally efficient alternative of generating the waveform frame-by-frame with large strides. We present SING, a lightweight neural audio synthesizer for the original task of generating musical notes given desired instrument, pitch and velocity. Our model is trained end-to-end to generate notes from nearly 1000 instruments with a single decoder, thanks to a new loss function that minimizes the distances between the log spectrograms of the generated and target waveforms. On the generalization task of synthesizing notes for pairs of pitch and instrument not seen during training, SING produces audio with significantly improved perceptual quality compared to a state-of-the-art autoencoder based on WaveNet [4] as measured by a Mean Opinion Score (MOS), and is about 32 times faster for training and 2, 500 times faster for inference.
Neural audio synthesis is a important and rapidly evolving field of study. Recent breakthroughs rely on expensive audio-sample level autoregressive models. The authors demonstrate a simpler frame-wise regression approach can do better than expected by generating raw audio, but then performing an STFT and regressing to targets in the log-spectral domain. This results in considerable speedups in inference time, as the sound can be generated in parallel. The authors wisely focus on NSynth as a controlled dataset to test conditional generation in a limited domain, and demonstrate generation that generalizes to unseen conditioning, generates audio that is low in distortion, and scores well on MOS tests.  Strengths: + STFTs as differentiable operations are a very promising direction. Despite being an obvious choice, sucessful examples of their use are relatively rare, and is the core of the novelty of this paper.   + First demonstration of low-distorion audio generated framewise with learned convolutional filters. Framewise filters are usually somewhat noisy when learned to generate raw audio. The combination of the spectral loss and the 'overlap-add' windowing are likely responsible, and demonstrating this is of broad value to those working with machine learning and audio.  + Conditional generation of pairs unseen pairs of conditioning demonstrates generalization of the model and the strong use of the conditioning variables. Forcing autoregressive models to utilize conditioning information is sometimes still a challenge as the model can sometimes learn to ignore them and rely on the autoregression for modeling dependencies.  Room for improvement: - The paper is a bit confused about its treatement of phase. The model uses a log power spectra as the regression loss function, which is invariant to phase values of local bins. The authors seem to indicate in L45-46 that this is good because audio perception is also invariant to phase, but this is the not the case. Audio perception is very sensitive to phase alignment from frame-to-frame and between harmonics in a single frame. Humans are relatively unaware of *constant phase offsets* applied to the whole spectrogram, which equate to a translation of the audio forward or backwards by a few milliseconds. The log-magnitude loss is also insensitive to constant phase offsets, but gives no information to the model as to how to realistically align the generated phases between frames and frequencies. This is very apparent in the generated audio, as indicated by a percieved lower frequency "warble" or "ringing". Autoregressive models handle the phase problem by working at the lowest possible stride, and it is clear that this model does not comperably handle phase. Section 3.2.1 also misrepresents the situation, by considering an example signal that is a single harmonic and a single frame. Even for a single harmonic, the value of the phase will precess at a constant rate in a stft porportional to the difference of frequency between the fft bin and the harmonic frequency. This is why the derivative of the unwrapped phase for a constant tone is a constant, giving the solid colors in the "Rainbowgrams" of the original NSynth paper. That said, the generated audio sounds much better than magnitudes with random phases. It would improve the paper greatly to more accurately describe the relationships of phase alignment and directly address phase as a limitation of the current technique. For example, presenting rainbowgrams of real, wavenet, and sing generated audio could highlight what aspects sing does well at, despite not receiving any phase error info during training, and what needs improvement. It would be good to include those graphs in the supplemental for all audio in the supplemental. Similarly, a baseline model that just estimates magnitudes for each frame and then uses Griffin-Lim to estimate phase would be very important for arguing that generating directly in the audio domain is an improvement for frame-wise generation. An extension to the current model, using log-magnitude and derivative of unwrapped phase as regression targets would perhaps generate more phase coherent audio.  - Reliance on temporal encodings reduces the generality of the proposed architecture. This requires the dataset to be preprocessed and well-aligned, which by design is the case for for the NSynth dataset.  - There are a few simple simple metrics that could be added to evaluations to make them more thorough. Since there are labels for the dataset, training a separate classifier could give classification accuracy (pitch, velocity, instrument). The authors could also use the network to compute Frechet Inception Distance (FID), as a proxy for "quality". It would allow for investigation of which pitches/velocities/instruments for which SING does better.  Small other comments: * The authors of the NSynth paper actually note that it takes 10 days on 32 K40 GPUs, which are significantly slower than P100 GPUs. (https://github.com/tensorflow/magenta/tree/master/magenta/models/nsynth). While the SING model still is much faster train it's probably more like a factor of 30-40 in reality. Either demphasizing the specific numbers of the cliam (not mentioning in the abstract), or coming up with a more quantitative measure (ex. FLOPS) would be more transparent.  * The text of the paragraph on L61 could be clearer. It could just be said that prior work focuses on speech and compositions of symbolic notes, while the current goal is learning timbre of individual notes.   * L94, the NSynth model can generate conditional waveforms (pitch, instrument, velocity) since it is an autoencoder (this is how the authors make a playable instrument with it). Also the baseline model generalized to generate pitches that were not seen at training. I think the distinction the authors are trying to make, that should be emphasized is that SING is a generative model, not an autoencoder, so requires no initial audio to condition on.  * L137, "spectal loss instead of a regression loss". The spectral loss _is_ a regression loss. I think the authors are trying to distinguish between regression to spectral coefficients and regression to wavefunction values.   * L193, "fasten", I think the authors mean to say "speed up" or "hasten"  * MOS of random mechanical turkers is perhaps less meaningful than in the case of speech, because while average people are experts at producing and listening to speech, the same cannot be said for music. For example, the phase artifacts present in SING might stand out quite strongly as unrealistic to a music producer or researcher, but less so to a casual listener who might associate "distortion" more with white noise or quantization errors. There's not a lot that the authors can do about that in the context of their experiments, but I just wanted to note, because it was strange to me that the MOS was similar for the SING and wavenet models, while to my ears the wavenet was far preferable most of the time.