Raytracer.jl: A Differentiable Renderer That Supports Parameter Optimization [pdf]
There is at least one fundamental problem with these kind of approaches : Working in pixel space make the problem non differentiable.<p>Imagine a simple scene with a single red point on white background. You are currently rendering a point on the left of your virtual screen and trying to update your camera parameters to match the target point at the right of the screen to register your camera parameters. Your gradients are zero. You are already screwed.<p>With more complex scenes, situation gets a little better, because you can get lucky, if you happen to have overlapping (between current and target) areas of similar colors then the gradient will tend to make them more overlapping and hopefully drag you towards the correct camera parameters.<p>Then you try to mitigate the initial fundamental problem with little tricks, to increase the likelihood that we are in the lucky configuration. You increase the blurriness to transform a single point in a little circle so you have more hope to have overlapping areas. Then you add some spatial pyramids (i.e. rendering with a scale of resolutions) to allow large movement across the screen instead of moving by one pixel by iteration. Then because it still often doesn&#x27;t work 
you add random restarting.<p>Among other issues are : -making object model non parametric makes the search harder.
 -making the rendering process too &quot;good&quot; makes the search harder (you are considering advanced lighting effects i.e. details before global picture). 
 -Even using triangles is already a bad choice, it was an optimization of the rendering pipeline to make the forward process faster, here we are tackling with the inverse problem so these kind of premature optimization are now playing against us.<p>So what are better approaches ? There are various approach.<p>The old school way, are key points registering methods. You render your scene you look for key-points (eventually dense) with their local statistics, and you match with key-points of the target, which allows you to jump almost directly towards the solution, that you finish aligning with Iterative Closest Point algorithms.<p>The new school way, you use an invertible GAN model to render an image. This gives you a latent vector for your images. You do the operation you want to do on this latent vector, and then use the generator forward to get a new image. This latent vector contain the information about the scene so you can train a neural network (eventually with attention) to answer all questions about your scene parameters from this latent parameters.