Solved by incpda in post #3 
              
              
                I finally managed to rasterize my model output before feeding it to the loss.backward() operation. 
The solution is kind of nasty and requires to initialize the coordinate representation x in the shape of the pixel-wise representation, with x values occupying only a fraction of this shape. Then the …