Solved by jpeg729 in post #2 
              
              
                Initial gradients equal to output - targets corresponds to 0.5 * MSELoss, not to L1Loss. 
If you differentiate MSELoss by hand you get 2 * (output - targets). 
If you differentiate L1Loss (= abs(output - targets)) with respect to output then you get either +1 or -1 depending on whether output - targ…