Answers.Body: <p>Tensorflow code for &quot;one-to-one&quot; style transfer:</p>
  <p><a href="https://github.com/phiana/speech-style-transfer-vae-gan-tensorflow" rel="nofollow noreferrer">https://github.com/phiana/speech-style-transfer-vae-gan-tensorflow</a></p>
  <p>it's the implementation of a 2021 paper. Speech style transfer, voice cloning or speech-to-speech synthesis are the keywords. Further research (looking at the state of the art) would yield some papers:</p>
  <ul>
  <li>MIST-Tacotron: End-to-End Emotional Speech Synthesis Using Mel-Spectrogram Image Style Transfer (<a href="https://ieeexplore.ieee.org/document/9726166" rel="nofollow noreferrer">https://ieeexplore.ieee.org/document/9726166</a>)</li>
  <li>Expressive Neural Voice Cloning (<a href="https://expressivecloning.github.io/" rel="nofollow noreferrer">https://expressivecloning.github.io/</a>)</li>
  </ul>
  <p>It seems to be a research area without much activity though. Maybe you could add something interesting :)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-14T19:36:00.267Z
Answers.Text: I tried to install the VAE-GAN out-of-the-box and it failed; also, for my purpose I need a many-to-many style transfer model. The MIST-Tacotron paper seems interesting but I couldn't find code or a repo. Expressive Neural Voice Cloning is interesting because it has both text-to-speech and speech-to-speech in the target voice.
Answers.UserDisplayName: emonigma
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-14T19:40:01.940Z
Answers.LastActivityDate: 2023-11-20T10:35:58.200Z
Answers.LastEditDate: 2023-11-20T10:35:58.200Z
Answers.UserDisplayName: Franco Marchesoni
Answers.Body: <p>Additional projects that might be of interest:</p>
  <ul>
  <li><a href="https://paperswithcode.com/paper/neural-voice-cloning-with-a-few-samples" rel="nofollow noreferrer">Neural Voice Cloning with a Few Samples - NeurIPS 2018 (Sercan O. Arik, Jitong Chen, Kainan Peng, Wei Ping, Yanqi Zhou)</a></li>
  </ul>
  <p>A neural voice cloning system is introduced, using a few audio samples to create personalized speech interfaces. Two approaches are explored: speaker adaptation, which fine-tunes a multi-speaker model with cloning samples, and speaker encoding, which trains a separate model to infer new speaker embeddings from cloning audios. Both methods achieve good performance in terms of speech naturalness and similarity to the original speaker. Although speaker adaptation offers better naturalness and similarity, speaker encoding demands less cloning time and memory, making it suitable for low-resource deployment.</p>
  <p>Here is an <a href="https://github.com/SforAiDl/Neural-Voice-Cloning-With-Few-Samples" rel="nofollow noreferrer">open-source implementation of the paper</a>, but the GitHub page says the project is archived since February 2021 and read-only.</p>
  <ul>
  <li><a href="https://github.com/CMsmartvoice/One-Shot-Voice-Cloning" rel="nofollow noreferrer">Unet-TTS: Improving Unseen Speaker and Style Transfer in One-shot Voice Cloning, arXiv:2109.11115 [cs.SD]</a></li>
  </ul>
  <p>In this paper, the authors present a novel one-shot voice cloning algorithm called Unet-TTS that has good generalization ability for unseen speakers and styles. Based on a skip-connected U-net structure, the new model can efficiently discover speaker-level and utterance-level spectral feature details from the reference audio, enabling accurate inference of complex acoustic characteristics as well as imitation of speaking styles into the synthetic speech. According to both subjective and objective evaluations of similarity, the new model outperforms both speaker embedding and unsupervised style modeling (GST) approaches on an unseen emotional corpus.</p>
  <p>The <a href="https://github.com/CMsmartvoice/One-Shot-Voice-Cloning/tree/master" rel="nofollow noreferrer">GitHub repo</a> has a link to a <a href="https://colab.research.google.com/drive/1sEDvKTJCY7uosb7TvTqwyUdwNPiv3pBW?usp=sharing" rel="nofollow noreferrer">Google Colab</a>.</p>
  <ul>
  <li><a href="https://beta.elevenlabs.io/" rel="nofollow noreferrer">ElevenLabs.io</a></li>
  </ul>
  <p>(Not Open Source, but has a free tier. Voice cloning becomes available in the Starter Tier, starting at 5$/month.)</p>
  <p>ElevenLabs initially built new text-to-speech models which rely on high compression and context understanding to render human speech ultra-realistically. Their tools aim to provide the necessary quality for voicing news, newsletters, books and videos. They also offer a suite of tools for <strong>voice cloning</strong> and designing synthetic voices.</p>
  <ul>
  <li><a href="https://beyondwords.io" rel="nofollow noreferrer">BeyondWords.io</a></li>
  </ul>
  <p>(Not Open Source, but has a free tier and is a partner of the Open Voice Network, a non-profit industry association dedicated to making voice technology worthy of user trust and it operates as a directed fund of The Linux Foundation.)</p>
  <p>Voice cloning is part of the <a href="https://beyondwords.io/pricing/#all-features" rel="nofollow noreferrer">enterprise plan</a> with custom pricing and requires 2-8 hours of recorded utterances following their script. See an <a href="https://www.youtube.com/watch?v=qzVihUH4-4g" rel="nofollow noreferrer">example of original and cloned voice in English on YouTube</a>. Although it sources non-English voices from partners such as Google and Amazon, it does not seem to support voice cloning in languages other than English.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-14T19:43:12.383Z
Answers.Text: Thanks! I didn't try the first, Neural Voice Cloning with a Few Samples. I tried the second, Unet-TTS, out-of-the-box and it fails with `ERROR: Could not find a version that satisfies the requirement tensorflow-gpu==2.6.0`. I tested ElevenLabs and liked the results very much, but unfortunately they don't have my target language. BeyondWords seems very good, but they require 2-8 hours of audio, which is infeasible in my case. So, if my use case had the luck of falling into ElevenLabs' language, I would use that, and so I'm accepting the answer.
Answers.UserDisplayName: emonigma
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-19T23:03:07.117Z
Answers.LastActivityDate: 2023-11-15T11:17:44.243Z
Answers.LastEditDate: 2023-11-15T11:17:44.243Z
Answers.UserDisplayName: Hans-Peter Schrei
Body: <p>I want to program and train a voice cloner, in part to learn about this area of AI, and in part to use as a prototype of audio for testing and getting feedback from early adopters before recording in a studio with voice actors. For the prototype, I have a set of recordings from voice actors. I would like to record my voice, in English or other languages, then run a neural network and produce an audio with the same text, intonation and emotion but with roughly the actors' voices. It doesn't need to be perfect; 80% right and believable would be enough to get good feedback and reach a final version of the script before recording. I have 30 minutes to one hour of utterances from each voice I want to clone.</p>
  <p>The closest I have found is Resemble.ai, which has an <a href="https://www.youtube.com/watch?v=f075EOzYKog" rel="nofollow noreferrer">impressive video</a>, but the public plan is only in English and other languages are prohibitively expensive. The engineer published a masters' thesis as an <a href="https://github.com/CorentinJ/Real-Time-Voice-Cloning" rel="nofollow noreferrer">open-source project</a>, but this project does only text-to-speech, not speech-to-speech. Another startup is <a href="https://play.ht/voice-cloning/" rel="nofollow noreferrer">play.ht</a>, but again it seems to be English-only.</p>
  <p><a href="https://github.com/andabi/deep-voice-conversion" rel="nofollow noreferrer">This open source project</a> seems to do what I want, cloning Kate Winslet's voice, but it has no installation instructions and so I haven't tried yet.</p>
  <p>Can you recommend an open-source project, ideally in Python and Tensorflow, to roughly replace a voice with another?</p>
  <p><em>Note</em>: This question is similar to <a href="https://ai.stackexchange.com/questions/15501/what-is-the-state-of-the-art-open-source-voice-cloning-tool-right-now">What is the State-of-the-Art open source Voice Cloning tool right now?</a> , except that that question is old and the project mentioned only does text-to-speech, not speech-to-speech.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-14T11:16:08.167Z
Comments.Text: @MahNeh A text-to-speech would remove most of the artistic effects from tone of voice, such as pauses, emphasis, cadence, irony, etc. We tried producing video with text-to-speech in standard voices and our users did not like it at all, so we need something with a more human touch.
Comments.UserDisplayName: emonigma
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-14T12:18:20.457Z
Comments.Text: @MahNeh Yes, I saw the forks, but as you say, they seem outdated. It may be a sign that I need to do a fork myself and keep it current!
Comments.UserDisplayName: emonigma
Tags: <neural-networks><tensorflow><speech-recognition><speech-synthesis><style-transfer>
Title: Open-source vocal cloning (speech-to-speech neural style transfer)
UserDisplayName: emonigma
ThreadText: # Open-source vocal cloning (speech-to-speech neural style transfer)
  
  (Asked by: username_0 on 2023-03-02)
  
  I want to program and train a voice cloner, in part to learn about this area of AI, and in part to use as a prototype of audio for testing and getting feedback from early adopters before recording in a studio with voice actors. For the prototype, I have a set of recordings from voice actors. I would like to record my voice, in English or other languages, then run a neural network and produce an audio with the same text, intonation and emotion but with roughly the actors' voices. It doesn't need to be perfect; 80% right and believable would be enough to get good feedback and reach a final version of the script before recording. I have 30 minutes to one hour of utterances from each voice I want to clone.
  
  The closest I have found is Resemble.ai, which has an [impressive video](https://www.youtube.com/watch?v=f075EOzYKog), but the public plan is only in English and other languages are prohibitively expensive. The engineer published a masters' thesis as an [open-source project](https://github.com/CorentinJ/Real-Time-Voice-Cloning), but this project does only text-to-speech, not speech-to-speech. Another startup is [play.ht](https://play.ht/voice-cloning/), but again it seems to be English-only.
  
  [This open source project](https://github.com/andabi/deep-voice-conversion) seems to do what I want, cloning Kate Winslet's voice, but it has no installation instructions and so I haven't tried yet.
  
  Can you recommend an open-source project, ideally in Python and Tensorflow, to roughly replace a voice with another?
  
  *Note*: This question is similar to [What is the State-of-the-Art open source Voice Cloning tool right now?](https://ai.stackexchange.com/questions/15501/what-is-the-state-of-the-art-open-source-voice-cloning-tool-right-now) , except that that question is old and the project mentioned only does text-to-speech, not speech-to-speech.
  
  **Tags:** neural-networks, tensorflow, speech-recognition, speech-synthesis, style-transfer
  
  > **Comments:**
  > **username_0**: @MahNeh A text-to-speech would remove most of the artistic effects from tone of voice, such as pauses, emphasis, cadence, irony, etc. We tried producing video with text-to-speech in standard voices and our users did not like it at all, so we need something with a more human touch.  
  > **username_0**: @MahNeh Yes, I saw the forks, but as you say, they seem outdated. It may be a sign that I need to do a fork myself and keep it current!  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-03-19. Score: 2 | Accepted answer
  
  Additional projects that might be of interest:
  
  * [Neural Voice Cloning with a Few Samples - NeurIPS 2018 (Sercan O. Arik, Jitong Chen, Kainan Peng, Wei Ping, Yanqi Zhou)](https://paperswithcode.com/paper/neural-voice-cloning-with-a-few-samples)
  
  A neural voice cloning system is introduced, using a few audio samples to create personalized speech interfaces. Two approaches are explored: speaker adaptation, which fine-tunes a multi-speaker model with cloning samples, and speaker encoding, which trains a separate model to infer new speaker embeddings from cloning audios. Both methods achieve good performance in terms of speech naturalness and similarity to the original speaker. Although speaker adaptation offers better naturalness and similarity, speaker encoding demands less cloning time and memory, making it suitable for low-resource deployment.
  
  Here is an [open-source implementation of the paper](https://github.com/SforAiDl/Neural-Voice-Cloning-With-Few-Samples), but the GitHub page says the project is archived since February 2021 and read-only.
  
  * [Unet-TTS: Improving Unseen Speaker and Style Transfer in One-shot Voice Cloning, arXiv:2109.11115 [cs.SD]](https://github.com/CMsmartvoice/One-Shot-Voice-Cloning)
  
  In this paper, the authors present a novel one-shot voice cloning algorithm called Unet-TTS that has good generalization ability for unseen speakers and styles. Based on a skip-connected U-net structure, the new model can efficiently discover speaker-level and utterance-level spectral feature details from the reference audio, enabling accurate inference of complex acoustic characteristics as well as imitation of speaking styles into the synthetic speech. According to both subjective and objective evaluations of similarity, the new model outperforms both speaker embedding and unsupervised style modeling (GST) approaches on an unseen emotional corpus.
  
  The [GitHub repo](https://github.com/CMsmartvoice/One-Shot-Voice-Cloning/tree/master) has a link to a [Google Colab](https://colab.research.google.com/drive/1sEDvKTJCY7uosb7TvTqwyUdwNPiv3pBW?usp=sharing).
  
  * [ElevenLabs.io](https://beta.elevenlabs.io/)
  
  (Not Open Source, but has a free tier. Voice cloning becomes available in the Starter Tier, starting at 5$/month.)
  
  ElevenLabs initially built new text-to-speech models which rely on high compression and context understanding to render human speech ultra-realistically. Their tools aim to provide the necessary quality for voicing news, newsletters, books and videos. They also offer a suite of tools for **voice cloning** and designing synthetic voices.
  
  * [BeyondWords.io](https://beyondwords.io)
  
  (Not Open Source, but has a free tier and is a partner of the Open Voice Network, a non-profit industry association dedicated to making voice technology worthy of user trust and it operates as a directed fund of The Linux Foundation.)
  
  Voice cloning is part of the [enterprise plan](https://beyondwords.io/pricing/#all-features) with custom pricing and requires 2-8 hours of recorded utterances following their script. See an [example of original and cloned voice in English on YouTube](https://www.youtube.com/watch?v=qzVihUH4-4g). Although it sources non-English voices from partners such as Google and Amazon, it does not seem to support voice cloning in languages other than English.
  
  > **Comments:**
  > **username_0**: Thanks! I didn't try the first, Neural Voice Cloning with a Few Samples. I tried the second, Unet-TTS, out-of-the-box and it fails with `ERROR: Could not find a version that satisfies the requirement tensorflow-gpu==2.6.0`. I tested ElevenLabs and liked the results very much, but unfortunately they don't have my target language. BeyondWords seems very good, but they require 2-8 hours of audio, which is infeasible in my case. So, if my use case had the luck of falling into ElevenLabs' language, I would use that, and so I'm accepting the answer.  
  
  ---
  
  ### Answer by username_1 on 2023-03-14. Score: 3
  
  Tensorflow code for "one-to-one" style transfer:
  
  <https://github.com/phiana/speech-style-transfer-vae-gan-tensorflow>
  
  it's the implementation of a 2021 paper. Speech style transfer, voice cloning or speech-to-speech synthesis are the keywords. Further research (looking at the state of the art) would yield some papers:
  
  * MIST-Tacotron: End-to-End Emotional Speech Synthesis Using Mel-Spectrogram Image Style Transfer (<https://ieeexplore.ieee.org/document/9726166>)
  * Expressive Neural Voice Cloning (<https://expressivecloning.github.io/>)
  
  It seems to be a research area without much activity though. Maybe you could add something interesting :)
  
  > **Comments:**
  > **username_0**: I tried to install the VAE-GAN out-of-the-box and it failed; also, for my purpose I need a many-to-many style transfer model. The MIST-Tacotron paper seems interesting but I couldn't find code or a repo. Expressive Neural Voice Cloning is interesting because it has both text-to-speech and speech-to-speech in the target voice.  
  
  ---
  
  