After some tests my custom BCE seems to work fine, even though the grads don’t exactly evolve the same way. Mine looks a bit more shaky, but nothing to really worry about I guess. However, when it comes to dice loss, even when I use it alone, it is a total catastrophy:

As you see, after 30 batches I see gradients exploding then totally collapsing to 0. After 60 batches they just become nan (there are more than 500 batches to process in theory). Maybe I should implement a backward on this one as it looks like it is not numerically stable. It is pretty annoying because my segmentation task involves small masks, therefore dice would be quite ideal to use combined with bce. Problem is I have not yet found where this problem comes from, as I am far from an expert in pytorch’s autograd behavior. It is also possible my optimizer is causing this mess.
EDIT: looks like a smaller learning rate is indeed helping. I still don’t know why I get nans though.