 Dear Fellow Scholars, this is two-minute papers with Karojona Ifeher. AI research has come a long, long way in the last few years. Not so long ago, we were lucky if we could train a neural network to understand traffic signs and since then, so many things happened. By harnessing the power of learning algorithms, we are now able to impersonate other people by using a consumer camera, generate high-quality virtual human faces for people that don't exist or pretend to be able to dance as a pro dancer by using an external video footage and transferring it onto ourselves. Even though we are progressing at a staggering pace, there is a lot of debate as to which research direction is the most promising going forward. Roughly speaking, there are two schools of thought. One, we recently talked about Richard Sutton's amazing article by the name The Better Lesson, in which he makes a great argument that AI research should not try to mimic the way the human brain works. He argues that instead, all we need to do is formulate our problems in a general manner so that our learning algorithm may find something that is potentially much better suited for a problem than our brain is. I put a link to this video in the description if you're interested. And two, a different school of thought says that we should look at all these learning algorithms that use a lot of powerful hardware and can do wondrous things like playing a bunch of Atari games at a superhuman level. However, they learn orders of magnitude slower than the human brain does, so it should definitely be worth it to try to study and model the human brain at least until we can match it in terms of efficiency. This school of thought is what we are going to talk about in this video. As an example, let's take a look at deep reinforcement learning in the context of playing computer games. This technique is a combination of a neural network that processes the visual data that we see on the screen and the reinforcement learner that comes up with the gameplay-related decisions. Absolutely amazing algorithm, a true breakthrough in AI research. Very powerful, however, also quite slow. And by slow, I mean that we can sit for an hour in front of our computer and wonder why our learner does not work at all because it loses all of its lives almost immediately. If we remain patient, we find out that it works, it just learns at a glacial pace. So why is this so slow? Well, two reasons. Reason number one is that the learning happens through incremental parameter adjustment. If a human fails really badly at a task, the human would know that a drastic adjustment to the strategy is necessary, while the deep reinforcement learner would start applying tiny, tiny changes to its behavior and test again if things got better. This takes a while and as a result seems unlikely to have a close relation to how we humans think. The second reason for it being slow is the presence of weak inductive bias. This means that the learner does not contain any information about the problem we have at hand or in other words, has never seen the game we are playing before and has no other previous knowledge about games at all. This is desirable in some cases because we can reuse one learning algorithm for a variety of problems. However, because this way, the AI has to test a stupendously large number of potential hypotheses about the game, we will have to pay for this convenience by using a mighty inefficient algorithm. But is this all really true? Does deep reinforcement learning really have to be so slow? And what on earth does this have to do with our brain? Well, this paper proposes an interesting counter argument that this is not necessarily true and argues that with a few changes, the efficiency of deep reinforcement learning may be drastically improved and get this. It also tells us that these changes are also possibly based in neuroscience. One such change is using episodic memory, which stores previous experiences to help estimating the potential value of different actions, and this way, drastic parameter adjustments become a possibility. And it not only improves the efficiency, but there is more to it because there are recent studies that show that using episodic memory indeed contributes to the learning of real humans and animals alike. And two, it is beneficial to let the AI implement its own reinforcement learning algorithm, a concept often referred to as learning to learn or met our reinforcement learning. This also helps obtaining more general knowledge that can be reused across tasks further improving the efficiency of the agent. Here you see a picture of an FMRI and some regions are marked with yellow and orange here. What could this possibly mean? Well, hold on to your papers because these highlight neural structures that implement a very similar metary reinforcement learning scheme within the human brain. It turns out that metary enforcement learning or this learning to learn scheme may not just be something that speeds up our AI algorithms, but maybe a fundamental principle of the human brain as well. So these two changes to the pre-enforcement learning not only drastically improve its efficiency, but it also suddenly maps quite a bit better to our brain. How cool is that? So which school of thought are you most fond of? Should we model the brain or should we listen to Richard Sutton's bitter lesson? Let me know in the comments. Also make sure to have a look at the paper. I found it to be quite readable and you really don't need to be a neuroscientist to read it and learn quite a few new things. Make sure to have a look at it in the video description. Now, I think you noticed that this paper doesn't contain the usual visual fireworks and is more complex than your average two-minute papers video and hence I expected to get significantly less views. That's not a great business model, but you know what? I made this channel so I can share with you all these important lessons that I learned during my journey. This has been a true privilege and I am thrilled that I am still able to talk about all these amazing papers without worrying too much whether any of these videos will go viral or not. This has only been possible because of your unwavering support on patreon.com slash two-minute papers. If you feel like chipping in, please click the Patreon link in the video description. And if you are more like a crypto person, we also support cryptocurrencies like Bitcoin, Ethereum and Litecoin, the addresses are also available in the video description. Thanks for watching and for your generous support and I'll see you next time.