Hi,
I have the same problem with the BCELoss.
I defined a customBCELoss using:
def customBCELoss(outputs, truths):
     loss_BCE = ( truths * torch.log(outputs) + 
                     torch.add(torch.neg(truths), 1.) * 
                     torch.log(torch.add(torch.neg(outputs), 1.)) )
    avg_BCE = torch.neg(torch.mean(loss_BCE))
    return avg_BCE

This custom loss gives the same results as the nn.BCELoss on all the batches for the first two epochs up to machine precision, but divergences on the third.
Does anyone know how to solve this problem ?