 Dear Fellow Scholars, this is two-minute papers with Dr. Karojona Ifehir. Today we are going to have a little taste of how smart an AI can be these days. And it turns out these new AI's are not only smart enough to solve some grade school math problems, but get this, a new development can perhaps even take a crack at university level problems. Is that even possible or is this science fiction? Well, the answer is yes, it is possible, kind of. So why kind of? Well, let me try to explain. This is opening as work from October 2021. The goal is to have their AI understand these questions, understand the mathematics, and reason about a possible solution for grade school problems. Hmm, all right. So this means that the GPT-3 AI might be suitable for the substrate of the solution. What is that? GPT-3 is a technique that can understand text, try to finish your sentences, even build websites, and more. So can it even deal with these test questions? Let's see it together. Hold on to your papers because in goes a grade school level question, a little math brain teaser if you will, and outcomes. My goodness, is that right? Here, outcomes not only the correct solution to the question, but even the thought process that led to this solution. Imagine someone claiming that they had developed an AI discapable 10 years ago. This person would have been logged into an asylum. And now it is all there, right in front of our eyes. Absolutely amazing. Okay, but how amazing. Well, it can't get everything right all the time and not even close. If we do everything right, we can expect it to be correct about 35% of the time. Not perfect, not even close, but it is an amazing step forward. So what is the key here? Well, yes, you guys did right. The usual suspects. A big neural network, and lots of training data. The key numbers are 175 billion model parameters, and it needs to read a few thousand problems and their solutions as training samples. That is a big rocket, and lots of rocket fuel, if you will. But this is nothing compared to what is to come. Now, believe it or not, here is a follow-up paper from just a few months later, January 2022, that claims to do something even better. This is not from OpenAI, but it piggybacks on OpenAI technology, as you will see in a moment. And this work promises that it can solve university-level problems. And when I saw this in the paper, I thought, really, now great school materials, okay, that is a great leap forward, but solving university-level math exams, that's where the gloves come off. I am really curious to see what this can do. Let's have a look together. Some of these brain teasers smell very much like MIT to me, surprisingly short and elegant questions that often seem much easier than they are. However, all of these require a solid understanding of fundamentals and sometimes even a hint of creativity. Let's see, yes, that is, indeed right, these are MIT introductory course questions. I love it. So can it answer them? Now, if we have been holding on to your paper so far, squeeze that paper, and let's see the results together. My goodness, these are all correct. Flying colors, perfect accuracy, at least on these questions. This is swift progress in just a few months. Absolutely amazing. So how is this black magic done? Yes, I know that's what you're waiting for, so let's pop the hood and look inside together. Mm-hmm. All right, two key differences from OpenAI's GPT-3-based solution. Difference number one. It gets additional guidance. For instance, it is told what topic we are talking about, what code library to reach out for, and what is the definition of mathematical concepts, for instance, what is a singular value decomposition. I would argue that this is not too bad. Students typically get taught these things before the exam, too. In my opinion, the key is that this additional guidance is done in an automated manner. The more automated, the better. Difference number two. The substrate here is not GPT-3, at least not directly, but codex. Codex is OpenAI's GPT language model that was fine-tuned to be excellent at one thing. And that is writing computer programs or finishing your code. And as we've seen in a previous episode, it really is excellent. For instance, it can not only be asked to explain a piece of code, even if it is written in assembly, or create a pong game in 30 seconds, but we can also give it plain text descriptions about a space game and it will write it. Codex is super powerful, and now it can be used to solve previously unseen university level math problems. Now that is really something. And it can even generate a bunch of new questions, and these are bona fide real questions. Not just exchanging the numbers, the new questions often require completely different insights to solve these problems. A little creativity I see well done little AI. So how good are these? Well, according to human evaluators, they are almost as good as the ones written by other humans. And thus, these can even be used to provide more and more training data for such an AI, more fuel for that rocket, and good kind of fuel. Excellent. And it doesn't end there. In the meantime, as of February 2022, scientists at OpenAI are already working on a follow-up paper that solves no less than high school mathematical-olimpiate problems. These problems require a solid understanding of fundamentals, proper reasoning, and often even that is not enough. Many of these tasks put up a seemingly impenetrable wall and climbing the wall typically requires a real creative spark. Yes, this means that this can get quite tough. And their new method is doing really well at these, once again, not perfect, not even close. But it can solve about 30 to 40% of these tasks, and that is a remarkable hit rate. Now we see that all of these works are amazing, and they all have their own trade-offs. They are good and bad at different things, and have different requirements. And most of all, they all have their own limitations. Thus, none of these works should be thought of as an AI that just automatically does human level math. What we see now is that there is swift progress in this area, and amazing new papers are popping up, not every year, but pretty much every month. And this is an excellent place to apply the first law of papers, which says that research is a process. Do not look at where we are, look at where we will be, two more papers down the line. So, what would you use this for? Please let me know in the comments below. I'd love to hear your ideas. And also, if you are excited by this kind of incredible progress in AR research, make sure to subscribe and hit the bell icon to not miss it when we cover these amazing new papers. This video has been supported by weights and biases. Look at this, they have a great community forum that aims to make you the best machine learning engineer you can be. You see, I always get messages from you fellow scholars telling me that you have been inspired by the series, but don't really know where to start. And here it is. In this forum, you can share your projects, ask for advice, look for collaborators, and more. Make sure to visit WNB.ME slash paper forum and say hi or just click the link in the video description. Our thanks to weights and biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.