I am not an expert, but according to this:  Make torch.nn.functional.multilabel_soft_margin_loss more stable #9141  and use logsigmoid at multilabel_soft_margin_loss to make it more stable  this loss was unstable. I don’t know which version of PyTorch you are using, but maybe you should try to use the latest release.