In this paper, we propose the generation of accented speech using generative adversarial networks (GANs). Through this work we make two main contributions a) The ability to condition latent representations while generating realistic speech samples b) The ability to efficiently generate long speech samples by using a novel latent variable transformation module that is trained using policy gradients. Previous methods are limited in being able to generate only relatively short samples or are not very efficient at generating long samples. The generated speech samples are validated through a number of various evaluation measures viz, a Wasserstein-GAN critic loss and through subjective scores on user evaluations against a competitive speech synthesis baseline. The evaluations demonstrate that the model generates realistic long speech samples conditioned on accent efficiently.
This paper presents a method for generating speech audio in a particular accent. The proposed approach relies on a generative adversarial network (GAN), combined with a policy approach for joining together generated speech segments. The latter is used to deal with the problem of generating very long sequences (which is generally difficult with GANs).
The problem of generating accented speech is very relevant since accent plays a large role in human communication and speech technology. Unfortunately, this paper is hard to follow. Some of the approach details are unclear and the research is not motivated well. The evaluation does not completely support the claims of the paper, e.g., there is no human judgment of whether the generated audio actually matches the desired accent.
Detailed comments, suggestions, and questions:
- It would be very useful to situate the research within work from the speech community. Why is accented modelling important? How is this done at the moment in speech synthesis systems? The paper gives some references, but without context. The paper from Ikeno and Hansen below might be useful.
- Accents are also a big problem in speech recognition (references below). Could your approach give accent-invariant representations for recognition?
- Figure 1: Add $x$, $y$, and the other variables you mention in Section 3 to the figure.
- What is $o$ in eq. (1)?
- Could you add a citation for eq. (2)? This would also help justifying that "it has a smoother curve and hence allows for more meaningful gradients".
- With respect to the critic $C_\nu$, I can see that it might be helpful to add structure to the hidden representation. In the evaluation, could you show the effect of having/not having this critic (sorry if I missed it)? The statement about "more efficient layers" is not clear.
- Section 3.4: If I understand correctly, this is a nice idea for ensuring that generated segments are combined sensibly. It would be helpful defining with "segments" refer to, and stepping through the audio generation process.
- Section 4.1: "using which we can" - typo.
- Section 5.1: "Figure 1 shows how the Wasserstein distance ..." I think you refer to the figure with Table 1?
- Figure 4: Add (a), (b) and (c) to the relevant parts in the figure.
References that might be useful:
- Ikeno, Ayako, and John HL Hansen. "The effect of listener accent background on accent perception and comprehension." EURASIP Journal on Audio, Speech, and Music Processing 2007, no. 3 (2007): 4.
- Van Compernolle, Dirk. "Recognizing speech of goats, wolves, sheep and… non-natives." Speech Communication 35, no. 1 (2001): 71-79.
- Benzeghiba, Mohamed, Renato De Mori, Olivier Deroo, Stephane Dupont, Teodora Erbes, Denis Jouvet, Luciano Fissore et al. "Automatic speech recognition and speech variability: A review." Speech communication 49, no. 10 (2007): 763-786.
- Wester, Mirjam, Cassia Valentini-Botinhao, and Gustav Eje Henter. "Are We Using Enough Listeners? No!—An Empirically-Supported Critique of Interspeech 2014 TTS Evaluations." In Sixteenth Annual Conference of the International Speech Communication Association. 2015.
The paper tries to address an important problem, and there are good ideas in the approach (I suspect Sections 3.3 and 3.4 are sensible). Unfortunately, the work is not presented or evaluated well, and I therefore give a week reject.