Solved by ptrblck in post #4 
              
              
                You would need to make step 3 differentiable, i.e. render the image through PyTorch (or a library built on top of PyTorch such as PyTorch3D which seems to implement a differentiable renderer) as I expect that Blender breaks the computation graph.