I think the important thing to note is that the larger one (for me at least) was the incorrect one. And it probably is the same in your case. So the easiest fix is to use your own custom loss function (which is what I did and it’s been working ever since).