Body: <p>I've been considering starting a project for some time on sound source identification.</p>
  <p>To be more specific, my goal is to be able to identify the &quot;sources&quot; for sound in videos. Moving parts clanging, lips speaking, hands clapping etc. I'd like to think a model trained to be able to do this might be helpful for:</p>
  <ul>
  <li>Identifying who is saying what in a crowd</li>
  <li>Discovering noise sources caught on video (ex. a carpenter's saw as he is talking to someone)</li>
  <li>Extending this to design a model for reading lips, to discern speech in silent video.</li>
  </ul>
  <p><a href="https://i.sstatic.net/EOs87.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/EOs87.png" alt="Helen Keller talking" /></a>
  (Taken from <a href="https://www.youtube.com/watch?v=8ch_H8pt9M8" rel="nofollow noreferrer">https://www.youtube.com/watch?v=8ch_H8pt9M8</a>)</p>
  <p>You might think of this task like grounding in NLP, expect for sound/speech instead. I'm sure this has been done before, and I'd like to conduct a literature review. So is there a name for this kind of sound-source identification?</p>
  <p>I've tried Googling &quot;Sound Source Identification&quot;, but it only returns Speech Classification results (Is this sound a car or a truck etc.)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-09T23:32:46.753Z
Comments.Text: There is "speaker identification", which is your first bullet point, although I don't know about the others. I think the model for reading lips would be separate from the rest of what you are trying to do however, since the rest of it can utilize the sound in the video.
Comments.UserDisplayName: Varun Vejalla
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-10T08:02:37.490Z
Comments.Text: Well to my knowledge, "speaker identification" is solely an audio problem: "Is the person currently talking someone I know, and if so who?".
Comments.UserDisplayName: madprogramer
Tags: <classification><computer-vision><audio-processing>
Title: Is there a problem for "Sound Source Identification in Video Footage"?
UserDisplayName: madprogramer
ThreadText: # Is there a problem for "Sound Source Identification in Video Footage"?
  
  (Asked by: username_0 on 2020-08-09)
  
  I've been considering starting a project for some time on sound source identification.
  
  To be more specific, my goal is to be able to identify the "sources" for sound in videos. Moving parts clanging, lips speaking, hands clapping etc. I'd like to think a model trained to be able to do this might be helpful for:
  
  * Identifying who is saying what in a crowd
  * Discovering noise sources caught on video (ex. a carpenter's saw as he is talking to someone)
  * Extending this to design a model for reading lips, to discern speech in silent video.
  
  [![Helen Keller talking](https://i.sstatic.net/EOs87.png)](https://i.sstatic.net/EOs87.png)
  (Taken from <https://www.youtube.com/watch?v=8ch_H8pt9M8>)
  
  You might think of this task like grounding in NLP, expect for sound/speech instead. I'm sure this has been done before, and I'd like to conduct a literature review. So is there a name for this kind of sound-source identification?
  
  I've tried Googling "Sound Source Identification", but it only returns Speech Classification results (Is this sound a car or a truck etc.)
  
  **Tags:** classification, computer-vision, audio-processing
  
  > **Comments:**
  > **username_1**: There is "speaker identification", which is your first bullet point, although I don't know about the others. I think the model for reading lips would be separate from the rest of what you are trying to do however, since the rest of it can utilize the sound in the video.  
  > **username_0**: Well to my knowledge, "speaker identification" is solely an audio problem: "Is the person currently talking someone I know, and if so who?".  
  
  ---
  
  ## 0 answers
  
  