TIMBRETRON: A WAVENET(CYCLEGAN(CQT(AUDIO))) PIPELINE FOR MUSICAL TIMBRE TRANSFER
In this work, we address the problem of musical timbre transfer, where the goal is to manipulate the timbre of a sound sample from one instrument to match another instrument while preserving other musical content, such as pitch, rhythm, and loudness. In principle, one could apply image-based style transfer techniques to a time-frequency representation of an audio signal, but this depends on having a representation that allows independent manipulation of timbre as well as highquality waveform generation. We introduce TimbreTron, a method for musical timbre transfer which applies "image" domain style transfer to a time-frequency representation of the audio signal, and then produces a high-quality waveform using a conditional WaveNet synthesizer. We show that the Constant Q Transform (CQT) representation is particularly well-suited to convolutional architectures due to its approximate pitch equivariance. Based on human perceptual evaluations, we confirmed that TimbreTron recognizably transferred the timbre while otherwise preserving the musical content, for both monophonic and polyphonic samples. We made an accompanying demo video 1 which we strongly encourage you to watch before reading the paper.
The paper proposes a method for converting recordings of a specific musical instrument to another. The proposed approach is apply CycleGAN, which was developed for image style transfer, to transfer spectrograms. The synthesis is done using WaveNet.
The paper is interesting in the core idea. It demonstrates that this combination of building blocks can indeed map recordings while achieving certain characteristics of the target instrument.
The paper correctly describes "timbre" as a catch-all term for characterizing instruments besides pitch and volume. The success of the method should be judged along two dimensions, which are both very subjective:
 - Does the method transfer "enough" of the target instrument's characteristics?
 - Is the resulting audio quality sufficient?
The paper is easy to follow for someone with background in signal processing. I believe it is sufficiently easy to follow for readers with general computer-science and machine-learning background.
The paper focusses a lot on the choice of spectral representation. It compares short-term Fourier transform (STFT), which is the generic standard, and Constant-Q Transforms (CQT), a variant of STFT that uses a logarithmic frequency axis. To someone with signal-processing background, the choice of CQT seems logical and not something that would be challenged strongly as long as a simple comparison to STFT confirms that it works a bit better. I find the comparison between the two too dominant in the paper, and distracting from the other issues that I feel are more important (see below). For example, Section 6.2 states "We demonstrate that the aforementioned translation does indeed result in a perceivable pitch shift when fed to our conditional WaveNet." But that is trivial: Since changing the playback sample rate by a few half steps does not fundamentally alter the perceived timbre of an instrument, and such a change will, by construction of CQT, shift the CQT representation, and since the WaveNet has seen examples of the source instrument for all notes, shifting the CQT respresentation must necessarily result in a perceived pitch change in the re-synthesized wveform that does not fundamentally change its timbre. The real question here is whether the reconstructed signal is of the same quality as, for example, a simple PSOLA-based pitch change would be.
A larger problem of the paper is that the result section seems to only test two instrument mappings, violin to flute, and piano to harpsichord. One notes that these instrument pairs mostly differ in spectral envelope, while they are rather similar in longer-term temporal variations, such as what is sometimes characterized as the ADSR curve (attack-decay-sustain-release) and vibrato. These, in my view, are very important aspects of a musical instrument's characteristics, which are not addressed by the paper. (Whether they are considered part of "timbre" is not clear, but without mapping these, one cannot meaningfully speak of mapping instruments, which is the end goal of this paper.)
Another big problem in my view is that the audio quality is just not good. I hear a lot of musical-noise artifacts and local timbre modulations. Also it is not clear why the source material is of poor quality (sounds quite noisy, most likely in part due to mu-law 8-bit encoding, and they sound like 11 kHz recordings), for which there is no justification in 2018.
Lastly, I am not happy with the "beam-search" approach. That approach is used to post-correct imperfections in the WaveNet synthesis. It samples multiple generation hypotheses, and re-weights hypotheses by how well they match the original CQT when converted back. The need for this indicates a fundamental flaw in the WaveNet synthesizer. The authors explicitly say they did not want to fix the WaveNet algorithm itself. In my view, this is what should have been done.
The authors should focus much more on how to achieve sufficient WaveNet synthesis quality. This should be the main bulk of the paper, and would be a requirement for me to accept the paper.
So overall, the paper feels a little too much of combining black boxes.
In terms of significance, I would not think that this paper is getting near solving this problem, hence I rate it of less significance in the current state of results.
Pros:
 - interesting idea
 - reasonable approach by combining existing building blocks
Cons:
 - too much focus on STFT vs. CQT
 - too little focus on getting WaveNet synthesis right
 - too limited experimental validation (too restricted choice of instruments)
 - poor resulting audio quality
 - feels too much of combining black boxes
As a result I rate the paper "not good enough" in its current form.