Oh I think I get it! When using dice, one pixel difference can make the loss go straight from 0 to 1. For instance, if the target is full background (so only zeros) and my prediction yields all zeros but one pixel is 1, then loss will be 1. However, changing that to full zero will directly put the loss at 0, which of course causes a very large gradient as it is totally nonlinear. That is in theory what the smooth variable is for, but it doesn’t seem to prevent it. I think I should try a larger value, this may very well help.