 Dear Fellow Scholars, this is two-minute papers with Dr. Karo Ejona Ifeher. Today, we are going to take a collection of photos like these and magically create a video where we can fly through these photos. And we are going to do all this with a twist. So, how is this even possible? Especially that the input is only a handful of photos. Well, typically we give it to a learning algorithm and ask it to synthesize a photorealistic video where we fly through the scene as we please. Of course, that sounds impassable. Especially that some information is given about the scene, but this is really not much. And as you see, this is not impassable at all. For the power of learning-based techniques, this previous AI is already capable of pulling off this amazing trick. And today, I am going to show you something even more incredible. Did you notice that most of these were shot during the daytime and these are all well-lit images? Every single one of them. So our question today is, can we perform view synthesis in the dark? And my initial answer would be a resounding no. Why? Well, in this case, we not only have to deal with less detail in the images, it would also be very difficult to stitch new views together if we have images like this one. Luckily, we have a choice. Instead, we can try something else and that is using raw sensor data instead. It looks like this. We get more detail, but uh-oh. Now we also have a problem. Do you see the problem here? Yes, that's right. In the raw sensor data, we have more detail, but also much more noise that contaminates this data too. So we either have to choose from less detail and less noise or from more detail more noise. So I guess that means that we get no view synthesis in the dark, right? Well, don't despair, not everything is lost yet. There are image denoising techniques that we can reach out to. Let's see if this gets any better. Hmm, it definitely got a lot better, but I have to be honest. This is not even close to the quality we need for view synthesis. And note that this one denoises a single image. A single image? Yes, finally, there is an opening here. Remember, in the world of nerves, we are not using a single image, we are using a package of images. A package contains much more information than just one image and hopefully it can be denoised better. So, this is what the previous method could do and now hold onto your papers and let's look at this new technique called Ronerf. Can it pull this off? Wow, seemingly it can. So now can we be greedy and hope that view synthesis works on this data? Let's see. My goodness, it really does. And we are not done yet, in fact we are just getting started. It can do even more. For instance, it can perform tone mapping on the underlying data to bring out even more detail from these dark images and here comes my favourite. Oh yes, we can also refocus these images and this highly sought after depth of field effects will start to appear. I love it. And what I love even more is that we can even play with this in real time to refocus the scene. This is a very impressive set of features, so let's take it out for a spin and marvel together at five amazing examples of what it can do. Yes, once again this is extremely noisy. For instance, can you read this street sign? Not a chance, right? And what about now? This looks like magic. I love it. Now let's start the view synthesis part and this looks really good given the noisy inputs. The previous original nerve technique could only produce this and this is not some ancient technique. Uh-uh. No sir, this is from just two years ago and today a couple of papers down the line and we get this. I can't believe it. We can even see the specular highlight moving around the edge of the car here. Outstanding. Two, actually let's have a closer look at specular highlights. Here is a noisy image, the denoise version and the view synthesis and the specular highlights are once again excellent. These are very difficult to capture because they change a great deal as we move the camera around and the photos are spaced out relatively far from each other. This means a huge challenge for the learning algorithm and as you see this one passes with flying colors. Three thin structures are always a problem. Look, an otherwise excellent previous technique had a great deal of trouble with the fence here even in a well-lit scene. So let's see, are you kidding me? doing the same with a bunch of nighttime photos there is not a chance that this will work. So let's see together. Look at that. I am out of words. Or you know what's even better, let's be really picky and look here instead, these areas are even more challenging and even these work really well. Such improvement in so little time. For as I am a light transport researcher by trade, I would love to look at it, resolve some more challenging specular highlights. For instance, you can see how the road reflects the streetlights here and the result looks not just passable. This looks flat out gorgeous. Now talking about gorgeous scenes, let's look at some more of those. Five putting it all together. This will be a stress test for the new technique. Let's change the viewpoint, refocus the scene and play with the exposure at the same time. That is incredible. What a time to be alive. And you are saying that it does all this from a collection of 25 to 200 photos. Today we can shoot these in seconds. Clearly, not even this technique is perfect, we can see that this does not match reality exactly, but going from a set of extremely noisy raw images to this is truly a sight to behold. The previous two-year-old technique couldn't even get close to these results. Bravo! And this is an excellent place for us to apply the first law of papers which says that research is a process. Do not look at where we are, look at where we will be, two more papers down the line. So, what do you think? What will we be able to do, two more papers down the line? And what would you use this for? Let me know in the comments below. Waits and biases provides tools to track your experiments in your deep learning projects. What you see here is their artifacts feature which speeds up the most common machine learning steps like uploading raw data, splitting it into training, validation and test sets, and of course, the best part starting to train a neural network. It is used by many prestigious labs including OpenAI, Toyota Research, GitHub and more. And the best part is that waits and biases is free for all individuals, academics and open source projects. Make sure to visit them through wnb.com slash papers or just click the link in the video description and you can get a free demo today. Our thanks to waits and biases for their long-standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.