Solved by PySimon in post #2 
              
              
                I changed loss.sum() to loss.mean() and the problem disappeared  I came up with this idea because the default reduction for MSELoss and L1Loss also seems to be “mean”. But I don’t really know why I get nan values when I use sum… Does somebody have an idea?