 Dear Fellow Scholars, this is Two Minute Papers with Dr. Karo Zsolnai-Fehir. Approximately a year ago, we talked about an absolutely amazing paper by the name VittuGame, in which we could grab a controller and become video game characters. It was among the first introductory papers to tackle this problem, and in this series, we always say that two more papers down the line and it will be improved significantly. So let's see what's in store and this time just one more paper down the line. This new work offers an impressive value proposition, which is to transform a real tennis match into a realistic looking video game that is controllable. This includes synthesizing not only movements, but also what effect the movements have on the ball as well. So how do we control this? And now hold on to your papers because we can specify where the next shot would land with just one click. For instance, we can place this red dot here. And now just think about the fact that this doesn't just change where the ball should go, but the trajectory of the ball should be computed using a physical model and the kind of shot the tennis player has to perform for the resulting ball trajectory to look believable. This physical model even contains the ball's spin velocity and the magnus effect created by the spin. The entire chain of animations has to be correct and that's exactly what happens here. With blue, we can also specify the position the player has to await in to hit the ball next. And these virtual characters don't just look like their real counterparts, they also play like them. You see the authors analyze the playstyle of these athletes and build a heat map that contains information about their usual shot placements for the forehand and backhand shots separately, the average velocities of these shots and even their favored recovery positions. If you have a closer look at the paper, you will see that they not only include this kind of statistical knowledge into their system, but they really went the extra mile and included common tennis strategies as well. So how does it work? Let's look under the hood. First, it looks at broadcast footage from which annotated clips are extracted that contain the movement of these players. If you look carefully, you see this red line on the spine of the player and some more. These are annotations that tell the AI about the pose of the players. It builds a database from these clips and chooses the appropriate piece of footage for the action that is about to happen, which sounds great in theory, but in a moment you will see that this is not nearly enough to produce a believable animation. For instance, we also need a rendering step which has to adjust this footage to the appropriate perspective as you see here. But we have to do way more to make this work. Look, without additional considerations, we get something like this. Not good. So what happened here? Well, given the fact that the source data sets contain matches that are several hours long, they therefore contain many different lighting conditions. With this, visual glitches are practically guaranteed to happen. To address this, the paper describes a normalization step that can even these changes out. How well does this do? It's job. Let's have a look. This is the unnormalized case. This short sequence appears to contain at least four of these glitches, all of which are quite apparent. And now, let's see the new system after the normalization step. Yep, that's what I'm talking about. But these are not the only considerations the authors had to take to produce these amazing results. You see, oftentimes, quite a bit of information is missing from these frames. Our season fellow scholars know not to despair because we can reach out to image-in-painting methods to address this. These can fill in missing details in images with sensible information. You can see NVIDIA's work from two years ago that could do this reliably for a great variety of images. These new work uses a learning-based technique called image-to-image translation to fill in these details. Of course, the advantages of this new system are visible right away and so are its limitations. For instance, temporal coherence could be improved, meaning that the tennis records can appear or disappear from one frame to another. The sprites are not as detailed as they could be, but none of this really matters. What matters is that now, what's been previously impossible is now possible and two more papers down the line, it is very likely that all of these issues will be ironed out. What a time to be alive! This episode has been supported by Lambda GPU Cloud. If you're looking for inexpensive Cloud GPUs for AI, check out Lambda GPU Cloud. They've recently launched Quadro RTX 6000, RTX 8000, and V100 instances, and hold onto your papers because Lambda GPU Cloud can cost less than half of AWS and Asia. Plus, they are the only Cloud service with 48GB RTX 8000. Join researchers at organizations like Apple, MIT, and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.com slash papers to sign up for one of their amazing GPU instances today. Thanks to Lambda for their long-standing support and for helping us make better videos for you. Thanks for watching and for your generous support, and I'll see you next time.