Could you try and simplify the problem to see if you still see the same behavior?

Change the model to a very basic thing (like one single Linear layer)
Change the loss to something simpler? (remove the absolute values)