Hi.
Changing the weight for the jacobian part to 100 makes it go to 0 nicely.
Similarly, putting it to a very low value makes the regular loss go to 0.
So it is an expression power of your model that fails to optimize both at the same time. Maybe try different activation? I am not sure that a relu network can do this task.