Sample Efficient Adaptive Text-to-Speech
We present a meta-learning approach for adaptive text-to-speech (TTS) with few data. During training, we learn a multi-speaker model using a shared conditional WaveNet core and independent learned embeddings for each speaker. The aim of training is not to produce a neural network with fixed weights, which is then deployed as a TTS system. Instead, the aim is to produce a network that requires few data at deployment time to rapidly adapt to new speakers. We introduce and benchmark three strategies: (i) learning the speaker embedding while keeping the WaveNet core fixed, (ii) fine-tuning the entire architecture with stochastic gradient descent, and (iii) predicting the speaker embedding with a trained neural network encoder. The experiments show that these approaches are successful at adapting the multi-speaker neural network to new speakers, obtaining state-of-the-art results in both sample naturalness and voice similarity with merely a few minutes of audio data from new speakers.
 This paper presents an approach to customize or adapt a text-to-speech synthesis system to a new speaker, given relatively small amount of data from that speaker. It is a very well written paper with rather strong results indicating high quality, naturalness, and similarity with real speech from a speaker can be achieved with the authors* proposed approach. I think the paper should be accepted for presentation at the conference. Few comments: a) In second equation in Section 2 authors state speaker identity “s” is part of conditioning inputs “h” but it is not shown in the Equation where “h” is replaced with “l, f_0” b) In related work, I think the speaker code work of Abdel-Hamid et al., e.g. Ossama Abdel-Hamid, Hui Jiang, “Fast speaker adaptation of hybrid NN/HMM model for speech recognition based on discriminative learning of speaker code,” ICASSP 2013 is worth citing. c) The result that synthesized speech performs better than real speech in speaker verification task is interesting. To me this points to a potential weakness in the verification methodology. Please comment if this may be the case.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Sample Efficient Adaptive Text-to-Speech
We present a meta-learning approach for adaptive text-to-speech (TTS) with few data. During training, we learn a multi-speaker model using a shared conditional WaveNet core and independent learned embeddings for each speaker. The aim of training is not to produce a neural network with fixed weights, which is then deployed as a TTS system. Instead, the aim is to produce a network that requires few data at deployment time to rapidly adapt to new speakers. We introduce and benchmark three strategies: (i) learning the speaker embedding while keeping the WaveNet core fixed, (ii) fine-tuning the entire architecture with stochastic gradient descent, and (iii) predicting the speaker embedding with a trained neural network encoder. The experiments show that these approaches are successful at adapting the multi-speaker neural network to new speakers, obtaining state-of-the-art results in both sample naturalness and voice similarity with merely a few minutes of audio data from new speakers.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper presents an approach to customize or adapt a text-to-speech synthesis system to a new speaker, given relatively small amount of data from that speaker. It is a very well written paper with rather strong results indicating high quality, naturalness, and similarity with real speech from a speaker can be achieved with the authors* proposed approach. I think the paper should be accepted for presentation at the conference. Few comments: a) In second equation in Section 2 authors state speaker identity “s” is part of conditioning inputs “h” but it is not shown in the Equation where “h” is replaced with “l, f_0” b) In related work, I think the speaker code work of Abdel-Hamid et al., e.g. Ossama Abdel-Hamid, Hui Jiang, “Fast speaker adaptation of hybrid NN/HMM model for speech recognition based on discriminative learning of speaker code,” ICASSP 2013 is worth citing. c) The result that synthesized speech performs better than real speech in speaker verification task is interesting. To me this points to a potential weakness in the verification methodology. Please comment if this may be the case.
assistant
 This paper presents an approach to customize or adapt a text-to-speech synthesis system to a new speaker, given relatively small amount of data from that speaker. It is a very well written paper with rather strong results indicating high quality, naturalness, and similarity with real speech from a speaker can be achieved with the authors* proposed approach. I think the paper should be accepted for presentation at the conference. Few comments: a) In second equation in Section 2 authors state speaker identity “s” is part of conditioning inputs “h” but it is not shown in the Equation where “h” is replaced with “l, f_0” b) In related work, I think the speaker code work of Abdel-Hamid et al., e.g. Ossama Abdel-Hamid, Hui Jiang, “Fast speaker adaptation of hybrid NN/HMM model for speech recognition based on discriminative learning of speaker code,” ICASSP 2013 is worth citing. c) The result that synthesized speech performs better than real speech in speaker verification task is interesting. To me this points to a potential weakness in the verification methodology. Please comment if this may be the case.