 Dear Fellow Scholars, this is 2 Minute Paper Sweet Carro Joel Naifa here. This is a neural network-based technique that can perform audio-visual separation. Before we talk about what that is, I will tell you what it is not. It is not what we've seen in the previous episode where we could select a pixel and listen to it. Have a look. And now let's try to separate the sound of the cello and see if it knows where it comes from. This one is different. This new technique can clean up an audio signal by suppressing the noise in a busy bar, even if the source of the noise is not seen in the video. It can also enhance the voice of the speaker at the same time. Let's listen. So this task is given the video, any person who gets cleaned up, and everything else gets suppressed. So this task is given the video, any person who gets cleaned up, and everything else gets suppressed. Or, if we have a Skype meeting with someone in a lab or a busy office where multiple people are speaking nearby, we can also perform a similar speed separation, which would be a gut send for future meetings. So we've been trying to train this network to input two embedding as an output of three. Yeah, this is just an extra experiment for the paper. Hi guys. And I think if you are a parent, the utility of this example needs no further explanation. Hi, my name is I'm a I am not sure if I ever encountered the term screaming children in the evening. I am not sure if I ever encountered the term screaming children in the abstract of an AI paper. So that one was also a first here. This is a super difficult task because the AI needs to understand what lip motions correspond to what kind of sounds, which is different for all kinds of languages, age groups, and head positions. To this end, the authors put together a stupendously large data set with almost 300,000 videos with clean speech signals. This data set is then run through a multi-stream neural network that detects the number of human faces within the video, generates small thumbnails of them, and observes how they move over time. It also analyzes the audio signals separately, then fuses these elements together with the recurrent neural network to output the separated audio waveforms. A key advantage of this architecture and training method is that as opposed to many previous works, this is speaker independent. Therefore, we don't need specific training data from the speaker we want to use this on. This is a huge leap in terms of usability. The paper also contains an excellent demonstration of this concept by taking a piece of footage from Conan O'Brien's show where two comedians were booked for the same time slot and talk over each other. The result is a performance where it is near impossible to understand what they are saying, but with this technique, we can hear both of them one by one crystal clear. You see some results over here, but make sure to click the paper link in the description to hear the sound samples as well. Thanks for watching and for your generous support and I'll see you next time.