 Dear Fellow Scholars, this is Two Minute Papers with Dr. Karajon Aifahir. Today, I will attempt to tell you a glorious tale about AI-based music generation. You see, there is no shortage of neural network-based methods that can perform physics simulations, style transfer, deepfakes, and a lot more applications where the training data is typically images or video. If the training data for a neural network is in pure text, it can learn about that. If the training data is waveforms and music, it can learn that too. Wait, really? Yes. In fact, let's look at two examples and then dive into today's amazing paper. In this earlier work, by the name, Look, Listen, and Learn, two scientists, a deep mind, set out to look at a large number of videos with sound. You see here that there is a neural network for processing the vision and one for the audio information. That sounds great, but what are these heatmaps? These were created by this learning algorithm, and they show us which part of the image is responsible for the sounds that we hear in the video. The harder the color, the more sounds are expected from a given region. It was truly amazing that it didn't automatically look for humans and colored them red in the heatmap. There are cases where the humans are expected to be the source of the noise, for instance, in concerts where, in other cases, they don't emit any noise at all. It could successfully identify these cases. This still feels like science fiction to me, and we covered this paper in 2017 approximately 250 episodes ago. You will see that we have come a long, long way since. We often say that these neural networks should try to embody general learning concepts. That's an excellent, and in this case, testable statement, so let's go ahead and have a look under the hood of this vision and audio processing neural networks, and, yes, they are almost identical. Some parameters are not the same because they have been adapted to the length and dimensionality of the incoming data, but the key algorithm that we run for the learning is the same. Later, in 2018, DeepMind published a follow-up work that looks at performances on the piano from the masters of the past and learns to play in their style. A key differentiating factor here was that it did not do what most previous techniques do, which was looking at the score of the performance. These older techniques knew what to play, but not how to play these notes, and these are the nuances that truly make music come alive. This method learned from raw audio waveforms and thus could capture much, much more of the artistic style. Let's listen to it, and in the meantime, you can look at the composers it has learned from to produce these works. However, in 2019, OpenAI recognized the text-based music synthesizers can not only look at a piece of score, but can also continue it, thereby composing a new piece of music, and what's more, they could even create really cool blends between genres. Listen, as their AI starts out from the first six notes of the Chopin piece and transitions into a pop style with a bunch of different instruments entering a few seconds in. Very cool. The score-based techniques are a little lacking in nuance, but can do magical, genre mixing, and more, whereas the waveform-based techniques are more limited, but can create much more sophisticated music. Are you thinking what I am thinking? Yes, you have guessed right. Hold on to your papers, because in OpenAI's new work, they tried to fuse the two concepts together, or, in other words, take a genre, an artist, and even lyrics as an input, and it would create a song for us. Let's marvel at a few curated samples together. The genre, artist, and lyrics information will always be on the screen. Wow, I am speechless. Love the AI-based lyrics, too. This has the nuance of waveform-based techniques with the versatility of score-based methods. Glorious. If you look in the video description, you will find a selection of uncurated music samples as well. It does what it does by compressing the raw audio waveform into a compact representation. In this space, it is much easier to synthesize new patterns after which we can decompress it to get the output waveforms. It has also learned to group up and cluster a selection of artists which reflects how the AI thinks about them. There is so much cool stuff in here that it would be worthy of a video of its own. Note that it currently takes 9 hours to generate 1 minute of music, and the network was mainly trained on western music, and only speaks English, but you know, as we always say around here, 2 more papers down the line, and it will be improved significantly. I cannot wait to report on them, should any follow-up works appear, so make sure to subscribe and hit the bell icon to not miss it. What a time to be alive. What you see here is an instrumentation of this exact paper we have talked about which was made by weights and biases. I think organizing these experiments really showcases the usability of their system. Weight and biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money, and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub, and more. And the best part is that if you are an academic or have an open source project, you can use their tools for free. It really is as good as it gets. Make sure to visit them through wnb.com slash papers, or just click the link in the video description, and you can get a free demo today. Our thanks to weights and biases for their long standing support, and for helping us make better videos for you. Thanks for watching, and for your generous support, and I'll see you next time.