 Dear Fellow Scholars, this is not two-minute papers with Dr. Karo Zonai Fahir, due to popular demand this is a surprise video with the talk of our new paper that we just published. This was the third and last paper in my PhD thesis and hence this is going to be a one-off video that is longer and a tiny bit more technical, I am keenly aware of it but I hope you'll enjoy it. Let me know in the comments when you have finished the video and worry not, all the upcoming videos are going to be in the usual two-minute papers format. The paper and the source code are all available in the video description. And now let's dive in. In a previous paper our goal was to populate this scene with over a hundred materials with a learning-based technique and create a beautiful planet with rich vegetation. The results looked like this. One of the key elements to accomplish this was to use a neural renderer or in other words the decoder network that you see here which took a material shader description as an input and predicted its appearance thereby replacing the renderer we used in the project. It had its own limitations for instance it was limited to this scene with a fixed lighting setup and only the material properties were subject to change. But in return it mimicked the global illumination renderer rapidly and faithfully. And in this new work our goal was to take a different vantage point and help artists with general image processing knowledge to perform material synthesis. Now this sounds a little nebulous so let me explain. One of the key ideas is to achieve this with a system that is meant to take images from its own renderer like the ones you see here. But of course we produce these ourselves so obviously we know how to do it so this is not very useful yet. However the twist is that we only start out with an image of this source material and then load it into a RESTOR image editing program like Photoshop and edit it to our liking and just pretend that this is achievable with our renderer. As you see many of these target images in the middle are results of poorly executed edits. For instance the stitched specular highlight in the first example isn't very well done and neither is the background of the gold target image in the middle. However in the next step our method proceeds to find a photorealistic material description that when rendered resembles this target image and works well even in the presence of these poorly executed edits. The whole process executes in 20 seconds. To produce a mathematical formulation for this problem we started with this. We have an input image t and edited to our liking to get the target image t with a tilt. Now we are looking for a shader parameter set x that when rendered with the phi operator approximates the edited image. The constraint below stipulates that we would remain within the physical boundaries for each parameter for instance albedos between 0 and 1 proper indices of refraction and so on. So how do we deal with phi? We used the previously mentioned neural renderer to implement it otherwise this optimization process would take 25 hours. Later we made an equivalent and constrained reformulation of this problem to be able to accommodate a greater set of optimizers. This all sounds great on paper and works reasonably well for materials that can be exactly matched with this shader like this one. This optimizer based solution can achieve it reasonably well. But unfortunately for more challenging cases as you see the target image on the lower right the optimizer's output leaves much to be desired. Note again that the result on the upper right is achievable with the shader while the lower right is a challenging imaginary material that we are trying to achieve. The fact that this is quite difficult is not a surprise because we have an only near and non-convex optimization problem which is also high dimensional. So this optimization solution is also quite slow but it can start inching towards the target image. As an alternative solution we also developed something that we call an inversion network this addresses the adjoint problem of neural rendering or in other words we show it the edited input image and outcomes the shader that would produce this image. We have trained 9 different neural network architectures for this problem which sounds great so how well did it work? Well we found out that none of them are really satisfactory for more difficult edits because all of the target images are far far outside of the training domain. We just cannot prepare the networks to be able to handle the rich variety of edits that come from the artist. However some of them are one could say almost usable for instance number one and five are not complete failures and note that these solutions are provided instantly. So we have two techniques none of them are perfect for our task a fast and approximate solution with the inversion networks and a slower optimizer that can slowly inch towards the target image. Our key insight here is that we can produce a hybrid method that fuses the two solutions together. The workflow goes as follows. We take an image of the initial source material and edit it to our liking to get this target image. Then we create a course prediction with a selection of inversion networks to initialize the optimizer with the prediction of one of these neural networks preferably a good one so the optimizer can start out from a reasonable initial guess. So how well does this hybrid method work? I'll show you in a moment here we start out with an achievable target image and then try two challenging image editing operations. This image can be reproduced perfectly as long as the inversion process works reliably. Unfortunately as you see here this is not the case. In the first row using the optimizer and the inversion networks separately we get results that fail to capture the specular highlight properly. In the second row we have deleted the specular highlight on the target image on the right and replaced it with a completely different one. I like to call this the Franken BRDF and it would be amazing if we could do this but unfortunately both the optimizer and the inversion networks flounder. Another thing that would be really nice to do is deleting the specular highlight and filling the image via image in painting. This kind of works with the optimizer but you'll see in a moment that it's not nearly as good as it could be. And now if you look carefully you see that our hybrid method outperforms both of these techniques in each of the three cases. In the paper we report results on a dozen more cases as well. We make an even stronger claim in the paper where we say that these results are close to the global optimum. You see the results of this hybrid method if you look at the intersection of now the meet and and then they are highlighted with the red ellipses. The records in the table show the RMS errors and are subject to minimization. With this you see that this goes neck and neck with the global optimizer which is highlighted with green. In summary our technique runs in approximately 20 seconds works for specular highlight editing, image blending, stitching, in painting and more. The proposed method is robust works even in the presence of poorly edited images and can be easily deployed in already existing rendering systems and allows for rapid material prototyping for artists working in the industry. It is also independent of the underlying principle shader so you can also add your own and expect it to work well as long as the neural renderer works reliably. A key limitation of the work is that it only takes images in this canonical scene with a carved sphere material sample but we can juxture that it can be extended to be more general and propose a way to do it in the paper. Make sure to have a closer look if you are interested. The teaser image of this paper is showcased in the 2020 computer graphics forum cover page. The whole thing is also quite simple to implement and we provide the source code, pre-trained networks on our website and all of them are under a permissive license. Thank you so much for watching this and the big thanks to Petavanka and Michaal Vima for advising this work.