 Dear Fellow Scholars, this is 2 Minute Papers with Dr. Karojejona Ifehir. Let's talk about video super resolution. The problem statement is simple, in goes a course video, the technique analyzes it, guesses what's missing, and out comes a detailed video. You know the CSI thing, and hence, however, of course, reliably solving this problem is anything that simple. This previous method is called TACOGAN and it is able to give us results that are often very close to reality. It is truly amazing how much this technique understands the world around us from just this training set of low and high resolution videos. However, as amazing as super resolution is, it is not the most reliable way of delivering high quality images in many real-time applications, for instance, video games. Note that in these applications, we typically have more data at our disposal, but in return, the requirements are also higher. We need high quality images, at least 60 times per second, temporal coherence is a necessity, or in other words, no jarring jumps and flickering is permitted. And hence, one of the techniques often used in these cases is called super sampling. At the risk of simplifying the term, super sampling means that we split every pixel into multiple pixels to compute a more detailed image and then display that to the user. And does this work? Yes, it does, it works wonderfully, but it requires a lot more memory and computation, therefore, it is generally quite expensive. So, our question today is, is it possible to use these amazing learning-based algorithms to do it a little smarter? Let's have a look at some results from a recent paper that uses a neural network to perform super sampling at a more reasonable computational cost. Now, in goes the low resolution input and my goodness, like magic, outcomes this wonderful, much more detailed result. And here is the reference, which is the true higher resolution image. Of course, the closer the neural super sampling is to this, the better. And as you see, this is indeed really close and much better than the pixelated inputs. Let's do one more. Wow! This is so close, I feel we are just a couple papers away from a result being indistinguishable from the real reference image. Now, we noted that this previous method has access to more information than the previously showcased super resolution method. It looks at not just one frame, but a few previous frames as well. Can use an estimation of the motion of each pixel over time and also gets depth information. This can be typically produced inexpensively with any major game engine. So how much of this data is required to train this neural network? Hold on to your papers because 80 videos were used and the training took approximately one and a half days on one Titan V, which is an expensive but commercially available graphics card. And no matter because this step only has to be done once and depending on how high the resolution of the output should be with the faster version of the technique, the up sampling step takes from 8 to 18 milliseconds, so this runs easily in real time. Now of course this is not the only modern super sampling method, this topic is subject to a great deal of research, so let's see how it compares to others. Here you see the results with TAAU, the temporal up sampling technique used in Unreal Engine and industry standard game engine. And look, this neural super sampler is significantly better at anti-aliasing or in other words, smoothing these jagged edges and not only that, but it also resolves many more of the intricate details of the image. Temporal coherence has also improved a great deal as you see the video output is much smoother for the new method. This paper also contains a pond more comparisons against recent methods, so make sure to have a look. Like many of us, I would love to see a comparison against Nvidia's DLSS solution, but I haven't been able to find a published paper on the later versions of this method. I remain very excited about seeing that too. And for now, the future of video game visuals and other real time graphics applications is looking as excited as it's ever been. What a time to be alive. This episode has been supported by weights and biases. In this post, they show you how to use their system to explore what exact code changes were made between two machine learning experiments. Weight and biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. The best part is that if you have an open source, academic or personal project, you can use their tools for free. It really is as good as it gets. Make sure to visit them through wnb.com slash papers or click the link in the video description to start tracking your experiments in five minutes. Our thanks to weights and biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.