Initial gradients equal to output - targets corresponds to 0.5 * MSELoss, not to L1Loss.
If you differentiate MSELoss by hand you get 2 * (output - targets).
If you differentiate L1Loss (= abs(output - targets)) with respect to output then you get either +1 or -1 depending on whether output - targets is positive or negative. The fact you see .25 rather than 1 in the actual gradients is because PyTorch takes the average loss per observation, rather than their sum.