 Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. In this new age of AI, there is no shortage of articles and discussion about AI safety and, of course, rightfully so. These new learning algorithms started solving problems that were previously thought to be impossible in quick succession. Only 10 years ago, if we told someone about half of the things that have been covered in the last few Two Minute Papers episodes would have been declared insane. And of course, having such powerful algorithms, we have to make sure that they are used for good. This work is a collaboration between open AI and deep-mind security team and is about introducing more human control in reinforcement learning problems. The goal was to learn to perform a backflip through reinforcement learning. This is an algorithm that tries to perform a series of actions to maximize the score, kind of like playing computer games. For instance, in Atari Breakout, if we break a lot of breaks, we get a high score, so we know we did something well. If we see that happening, we keep doing what led to this result. If not, we go back to the drawing board and try something new. But this work is about no ordinary reinforcement learning algorithm because the score to be maximized comes from a human supervisor and we are trying to teach a digital creature to perform a backflip. I particularly like the choice of the backflip here because we can tell when we see one, but the mathematical specification of this in terms of movement actions is rather challenging. This is a problem formulation in which humans can overlook and control the learning process, which is going to be an increasingly important aspect of learning algorithms in the future. The feedback option is very simple, we just specify whether the sequence of motions achieved our prescribed goal or not. Did it fall or did it perform the backflip successfully? After around 700 human feedbacks, the algorithm was able to learn the concept of a backflip, which is quite remarkable given that these binary yes-no scores are extremely difficult to use for any sort of learning. In an earlier episode, we illustrated a similar case with a careless teacher who refuses to give out points for each problem on a written exam and only announces whether we have failed or passed. This clearly makes a dreadful learning experience and it is incredible that the algorithm is still able to learn using these. We provide feedback on less than 1% of the actions the algorithm makes and it can still learn difficult concepts off of these extremely sparse and vague rewards. Low quality teaching leads to high quality learning. How about that? This is significantly more complex than what other techniques were able to learn with human feedback and it works with other games too. A word about the collaboration itself. When a company hires a bunch of super smart scientists and spends a ton of money on research, it is understandable that they want to get an edge through these projects which often means keeping the results for themselves. This leads to excessive secrecy and the lack of collaboration with other groups as everyone wants to keep their cards close to their chest. The fact that such collaborations can happen between these two AR research giants is a testament to how devoted they are to working together and sharing their findings with everyone free of charge for the greater good. Awesome. As the media is all up in arms about the demise of the human race, I feel that it is important to show the other side of the coin as well. We have top people working on AI safety right now. If you wish to help us tell these stories to more and more people, please consider supporting us on Patreon. Details are available in the video description or just click the letter P that appears on the screen in a moment. Thanks for watching and for your generous support and I'll see you next time.