WaveNet: A Generative Model for Raw Audio
This can be used to implement seamless voice performance transfer from one speaker to another:<p>1. Train a WaveNet with the source speaker.<p>2. Train a second WaveNet with the target speaker. Or for something totally new, train a WaveNet with a bunch of different speakers until you get one you like. This becomes the <i>target WaveNet</i>.<p>3. Record raw audio from the source speaker.<p>Fun fact: any algorithmic process that "renders" something given a set of inputs can be "run in reverse" to recover those inputs given the rendered output. In this case, we now have raw audio from the source speaker that—in principle— <i>could</i> have been rendered by the source speaker's WaveNet, and we want to recover the inputs that would have rendered it, had we done so.<p>To do that, usually you convert all numbers in the forward renderer into Dual numbers and use automatic differentiation to recover the inputs (in this case, phonemes and what not).<p>4. Recover the inputs. (This is computationally expensive, but not difficult in practice, especially if WaveNet's generation algorithm is implemented in C++ and you've got a nice black-box optimizer to apply to the inputs, of which there are many freely available options.)<p>5. Take the recovered WaveNet inputs, feed them into the target speaker's WaveNet, and record the resulting audio.<p>Result: The resulting raw audio will have the same overall performance and speech as the source speaker, but rendered completely naturally in the target speaker's voice.