How is the quality of singing voice measured?
Automatic: Normalized cross correlation (NCC)
Manual: Mean Opinion Score (MOS)
Singing voice conversion is to convert a singer's voice to another one's voice without changing singing content. Recent work shows that unsupervised singing voice conversion can be achieved with an autoencoder-based approach [1]. However, the converted singing voice can be easily out of key, showing that the existing approach can not model the pitch information precisely. In this paper, we propose to advance the existing unsupervised singing voice conversion method proposed in [1] to achieve more accurate pitch translation and flexible pitch manipulation. Specifically, the proposed PitchNet added an adversarially trained pitch regression network to enforce the encoder network to learn pitch invariant phoneme representation, and a separate module to feed pitch extracted from the source audio to the decoder network. Our evaluation shows that the proposed method can greatly improve the quality of the converted singing voice (2.92 vs 3.75 in MOS). We also demonstrate that the pitch of converted singing can be easily controlled during generation by changing the levels of extracted pitch before passing it to the decoder network.
Singing is an important way of human expression and the techniques of singing synthesis have broad applications in different prospects including virtual human, movie dubbing and so on. Traditional singing synthesize systems are based on concatenative BIBREF1 or HMM BIBREF2 based approaches. With the success of deep learning in Text-to-Speech, some neural singing synthesis methods have also been proposed recently. For example, BIBREF3 introduces a singing synthesis method using an architecture similar to WaveNet BIBREF4. It adopts lyrics and notes as input and generates vocoder features autoregressively for final singing voice synthesis. Singing voice conversion is another way of singing synthesis which extracts musical expression within existing singing and reproduced them with another singer's voice. It is very similar to speech based voice conversion BIBREF5, BIBREF6, BIBREF7, BIBREF8, but compared with speech voice conversion, singing voice conversion needs to deal with a wider range of frequency variations as well as the sharper change of volume and pitch within singing voice. The performance of singing conversion is highly dependent on the musical expression of converted singing and the similarity of the converted voice timbre compared to target singer's voice. There are several singing voice conversion methods to convert one's singing voice to another BIBREF9, BIBREF10, BIBREF11. They generally require parallel data to train the conversion model. To overcome the limitation of the parallel training data for singing voice conversion, an unsupervised method BIBREF0 has been proposed to utilize non-parallel data. This method employs an autoencoder architecture composed of a WaveNet-like encoder, a WaveNet BIBREF4 autoregressive decoder, and a learnable singer embedding table. Voice waveform is passed into the encoder and the output of the encoder will be concatenated with the embedding vector associated with the singer. The concatenated features will be used to condition the WaveNet decoder to reconstruct the input audio. A confusion loss BIBREF12 is also introduced to force the encoder to learn a singer-invariant representation. By switching between embeddings of different singers during generation, the singing voice conversion can be achieved. While this approach could generates singing voice perceptually similar to target singer, the quality of generated singing often suffers due to the difficulty on learning joint representation of phonetic and pitch representation. To address the difficulty of learning join phonetic and pitch representation in BIBREF0, we propose to use adversarially trained pitch regression network to encourage the encoder network to learn not only singer-invariant but also pitch-invariant representation, at the same time extract the pitch from source audio to be used as an additional input to the decoder. The proposed method can greatly improve the quality of the converted voice and achieve flexible pitch manipulation at the same time. In the following sections, we will introduce our proposed method in section SECREF2. And then section SECREF3 will show that our method is effective by quantitative and qualitative experiments. Finally, we will conclude in section SECREF4 and acknowledgements are in SECREF5.