Solved by PySimon in post #3 
              
              
                Okay, another update. I found the problem. It was indeed the error function… I was using the standard reduction mode which averages the losses. Using this instead solves it: 
lossfct = torch.nn.L1Loss(reduction='none')
[...]
for l in loss:
    l.mean().backward(retain_graph=True)

instead of 
lossfc…