 Dear Fellow Scholars, this is Two Minute Papers with Dr. Karo Zsolnai-Fehir. Today, we are going to take a light simulation algorithm that doesn't work too well and infuse it with a powerful neural network. And what happens? Then, and this happens. Wow! So, what was all this about? Well, when you fire up a classical light simulation program, first we start out from, yes, absolutely nothing. And look, over time, as we simulate the path of more and more light rays, we get to know something about the scene. And slowly, over time, the image cleans up. But, there is a problem. What is the problem? Well, look, the image indeed cleans up over time. But, no one said that this process would be quick. In fact, this can take from minutes in the case of this scene to get this even days for this scene. In our earlier paper, we rendered this beautiful, but otherwise, sinfully difficult scene and it took approximately three weeks to finish. And it also took several computers running at the same time. However, these days, neural network-based learning methods are already capable of doing light transport. So, why not use them? Well, yes, they are, but they are not perfect. And we don't want an imperfect result. So, scientists at Nvidia said that this might be the perfect opportunity to use control variants. What are those? And why is this the perfect opportunity? Control variants are a way to inject our knowledge of a scene into the light transport algorithm. And here's the key. Any knowledge is useful as long as it gives us a head start, even if this knowledge is imperfect. Here is an illustration of that. What we do is that we start out using that knowledge and make up for the differences over time. Okay, so how much of a head start can we get with this? Normally, we start out from a black image and then a very noisy image. Now, hold on to your papers and let's see what this can do to help us. Wow! Look at that! This is incredible. What you're looking at is not the final rendering. This is what the algorithm knows. And instead of the blackness, we can start out from this. Goodness, it turns out that this might not have been an illustration, but the actual knowledge the AI has of the scene. So cool! Let's look at another example. In this bedroom, this will be our starting point. My goodness! Is that really possible? The bed and the floor are almost completely done. The curtain and the refractive objects are noisy, but do not worry about those for a second. This is just a starting point and it is still way better than starting out from a black image. To be able to visualize what the algorithm has learned with a little more finesse, we can also pick a point in space and learn what the world looks like from this point and how much light scatters around it. And we can even visualize what happens when we move this point around. Man, this technique has a ton of knowledge about these scenes. So once again, just to make sure we start out not from a black image, but from a learned image and now we don't have to compute all the light transport in the scene we just need to correct the differences. This is so much faster, but actually is it? Now let's look at how this helps us in practice. And that means, of course, equal time comparisons against previous light transport simulation techniques. And when it comes to comparisons, you know what I want? Yes, I want Eric Vitch's legendary scene. See, this is an insane scene where all the light is coming from not here, but the neighboring room through a door that is just slightly a jar. And the past tracer, the reference light transport simulation technique behaves as expected. Yes, we get a very noisy image because very few of the simulated rays make it to the next room. Thus, most of our computation is going to waste. This is why we get this noisy image. And let's have a look what the new method can do in the same amount of time. Wow! Are you seeing what I am seeing? This is completely out of this world. My goodness! But this was a pathological scene designed to challenge our light transport simulation algorithms. What about a more typical outdoor scene? Add tons of incoming light from every direction, and my favorite water caustics. Do we get any advantages here? Hmm, the past tracer is quite noisy. This will take quite a bit of time to clean up. Whereas the new technique, oh my, that is a clean image. How close is it to the fully converged reference image? Well, you tell me because you are already looking at that. Yes, now we are flicking between the new technique and the reference image, and I can barely tell the difference. There are some, for instance, here, but that seems to be about it. Can you tell the difference? Let me know in the comments below. Now, let's try a bathroom. Lots of shiny surfaces and specular light transport. And the results are... Look at that! There is no contest here. A huge improvement across the whole scene. And believe it or not, you still haven't seen the best results yet. Don't believe it. Now, if you have been holding onto your papers, squeeze that paper and look at the art room example here. This is an almost unusable image with classical light transport. And are you ready? Well, look at this. What in the world? This is where I fell off the chair when I was reading this paper. Absolute madness. Look, while the previous technique is barely making a dent into the problem, the new method is already just a few fireflies away from the reference. I can't believe my eyes. And this result is not just an anomaly. We can try a kitchen scene and draw similar conclusions. Let's see. Now we're talking. I am out of words. Now, despite all these amazing results, of course, not even this technique is perfect. This tourist was put inside a glass container and is a nightmare scenario for any kind of light transport simulation. The new method successfully harnesses our knowledge about the scene and accelerates the process a great deal. But once again, we get fireflies. These are going to be difficult to get rid of and will still take a fair bit of time. But my goodness, if this is supposed to be a failure case, then yes, sign me up right now. Once again, the twist here is not just to use control variants, the initial knowledge thing, because in and of itself, it is not new. I, like many others, have been experimenting with this method back in 2013, almost 10 years ago, and back then, it was nowhere near as good as this one. So, what is the twist then? The twist is to use control variants and infuse them with a modern neural network. Infusing previous techniques with powerful learning-based methods is a fantastic area of research these days. For instance, here you see an earlier result, an ancient light transport technique called radiosity. This is what it was capable of back in the day. And here is the neural network-infused version. Way better. I think this area of research is showing a ton of promise and I'm so excited to see more in this direction. So, what do you think? What would you use this for? I'd love to hear your thoughts, please let me know in the comments below. And when watching all these beautiful results, if you feel that this light transport thing is pretty cool, and you would like to learn more about it, I held a master-level course on this topic at the Technical University of Vienna. Since I was always teaching it to a handful of motivated students, I thought that the teachings shouldn't only be available for the privileged few who can afford a college education, but the teachings should be available for everyone. Free education for everyone, that's what I want. So, the course is available free of charge for everyone, no strings attached, so make sure to click the link in the video description to get started. We write a full-light simulation program from scratch there and learn about physics, the world around us, and more. If you watch it, you will see the world differently. This video has been supported by weights and biases. Look at this, they have a great community forum that aims to make you the best machine learning engineer you can be. You see, I always get messages from you fellow scholars telling me that you have been inspired by the series, but don't really know where to start. And here it is, in this forum you can share your projects, ask for advice, look for collaborators and more. Make sure to visit www.me-slash-paper-forum and say hi or just click the link in the video description. Our thanks to weights and biases for their long-standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.