Body: <p><strong>Background:</strong></p>
  <p>I am working on a research project to use (demonstrate) the possibilities of Machine Learning and AI in artistic projects. One thing we are exploring is demonstrating deep fakes on stage. Of course, a deep fake is not easy to make. Still, we are exploring creating a &quot;minor quality&quot; deep fake live on a stage (or maybe in some other ways where people can make deep fakes of themselves) in which we put words into someone's mouth. I discovered that a semi-nice deep fake of the facial movement is possible, now a also want to add voice.</p>
  <p>There are a lot of text-to-speech systems, which allow using a voice that is created from the recording of the voice of a real person. That is already nice.</p>
  <p>The video is based on the facial movements of another person. So the Audio has to match the facial movements. The easiest way would be, if the &quot;fake&quot; voice says it exactly the same way, as the person doing the &quot;facial acting&quot; for the deep fake said it.</p>
  <p><strong>Question:</strong></p>
  <p>Is it possible to do a fake voice of a person in this way:</p>
  <ol>
  <li>Another person (the actor, or source) speaks the words in his voice and records it.</li>
  <li>The person with the faked voice (the destination) gives a voice sample with some random spoken text.</li>
  <li>An AI/algorithm/whatever modifies the recording from (1) in such a way, that the tone/voice is modified so it matches the voice from (2).</li>
  </ol>
  <p>Do systems/research like this exist? I did not find anything using google, but maybe I did not use the correct keywords.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-18T13:39:35.943Z
Comments.Text: I think this is probably possible. The research I'm aware of is about images. Have a look at CycleGAN, which learns to convert horses into zebras, with just a bunch of horse pictures and an unrelated bunch of zebra pictures.
Comments.UserDisplayName: Criticize SE actions means ban
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-18T13:40:52.053Z
Comments.Text: also StyleGAN which transfers the "artistic style" from one picture onto another picture. Voice is just "artistic style" but for sound.
Comments.UserDisplayName: Criticize SE actions means ban
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-18T13:41:12.317Z
Comments.Text: of course there is no way to know for sure without actually trying it and probably trying a bunch of dead ends before it might actually work
Comments.UserDisplayName: Criticize SE actions means ban
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-18T14:07:53.223Z
Comments.Text: @user253751 Thanks for your input! The repos you are listing are also interesting. Unfortunately this is out of my skill and time possibilities.
Comments.UserDisplayName: Nathan
Tags: <natural-language-processing><reference-request><deepfakes><speech-synthesis>
Title: Is Speech to Speech with changing the voice to a given other voice possible?
UserDisplayName: Nathan
ThreadText: # Is Speech to Speech with changing the voice to a given other voice possible?
  
  (Asked by: username_0 on 2021-09-18)
  
  **Background:**
  
  I am working on a research project to use (demonstrate) the possibilities of Machine Learning and AI in artistic projects. One thing we are exploring is demonstrating deep fakes on stage. Of course, a deep fake is not easy to make. Still, we are exploring creating a "minor quality" deep fake live on a stage (or maybe in some other ways where people can make deep fakes of themselves) in which we put words into someone's mouth. I discovered that a semi-nice deep fake of the facial movement is possible, now a also want to add voice.
  
  There are a lot of text-to-speech systems, which allow using a voice that is created from the recording of the voice of a real person. That is already nice.
  
  The video is based on the facial movements of another person. So the Audio has to match the facial movements. The easiest way would be, if the "fake" voice says it exactly the same way, as the person doing the "facial acting" for the deep fake said it.
  
  **Question:**
  
  Is it possible to do a fake voice of a person in this way:
  
  1. Another person (the actor, or source) speaks the words in his voice and records it.
  2. The person with the faked voice (the destination) gives a voice sample with some random spoken text.
  3. An AI/algorithm/whatever modifies the recording from (1) in such a way, that the tone/voice is modified so it matches the voice from (2).
  
  Do systems/research like this exist? I did not find anything using google, but maybe I did not use the correct keywords.
  
  **Tags:** natural-language-processing, reference-request, deepfakes, speech-synthesis
  > **Comments:**
  > **username_1**: I think this is probably possible. The research I'm aware of is about images. Have a look at CycleGAN, which learns to convert horses into zebras, with just a bunch of horse pictures and an unrelated bunch of zebra pictures.  
  > **username_1**: also StyleGAN which transfers the "artistic style" from one picture onto another picture. Voice is just "artistic style" but for sound.  
  > **username_1**: of course there is no way to know for sure without actually trying it and probably trying a bunch of dead ends before it might actually work  
  > **username_0**: @user253751 Thanks for your input! The repos you are listing are also interesting. Unfortunately this is out of my skill and time possibilities.  
  
  ---
  
  ## 0 answers
  
  