ptrblck:

Is a negative loss expected in your case?


I am building a model for semantic segmentation, and the loss function is a combination of focal and Dice loss. Dice loss cannot be negative and neither can focal loss(as BCE would be between 0 and 1) so I dont think it should be negative(right?).
The main issue was that I was not observing consistent behavior between FP32 and FP16 which has been resolved now,(due to that optimizer.zero_grad. ). If I still get the same loss, It can be attributed to the network itself(like always giving output as zero due to dying relu. )
Thanks a lot 
Update - There is one thing though, and I am not able to justify this - In the following code for every epoch  I check whether the weights are same even after optim.step
The code is -
image = torch.randn((1, 3, 640, 640)).cuda()
mask = torch.randint(0, 2, (1, 640, 640)).cuda().float()

model = nn.Sequential(nn.Conv2d(in_channels=3, out_channels=3, kernel_size=3, stride=1, padding=1),
                       nn.Conv2d(in_channels=3, out_channels=1, kernel_size=1, stride=1, padding=0))
model = model.cuda().train()

optim = torch.optim.Adam(model.parameters(), lr=1e-3)
use_amp = True
loss_list = []

for i in range(10):
    #optim.zero_grad()
    a1 = list(model.parameters())[0].clone()
    a2 = list(model.parameters())[1].clone()
    a3 = list(model.parameters())[2].clone()
    a4 = list(model.parameters())[3].clone()
    with torch.cuda.amp.autocast(enabled=use_amp):
        out = model(image)
        l = loss(out, mask)
    if use_amp:
        scaler.scale(l).backward()
        scaler.step(optim)
        scaler.update()
    else:
        l.backward()
        optim.step()
    print(l)
    b1 = list(model.parameters())[0].clone()
    b2 = list(model.parameters())[1].clone()
    b3 = list(model.parameters())[2].clone()
    b4 = list(model.parameters())[3].clone()
    print(torch.equal(a1, b1), torch.equal(a2, b2), torch.equal(a3, b3), torch.equal(a4, b4))
    print("-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=")


And interestingly the output is -
tensor(4.0811, device='cuda:0', grad_fn=<AddBackward0>)
True True True True
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
tensor(4.0811, device='cuda:0', grad_fn=<AddBackward0>)
True True True True
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
tensor(4.0811, device='cuda:0', grad_fn=<AddBackward0>)
True True True True
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
tensor(4.0811, device='cuda:0', grad_fn=<AddBackward0>)
True True True True
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
tensor(4.0811, device='cuda:0', grad_fn=<AddBackward0>)
True True True True
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
tensor(4.0811, device='cuda:0', grad_fn=<AddBackward0>)
True True True True
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
tensor(4.0811, device='cuda:0', grad_fn=<AddBackward0>)
True True True True
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
tensor(4.0811, device='cuda:0', grad_fn=<AddBackward0>)
True True True True
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
tensor(4.0811, device='cuda:0', grad_fn=<AddBackward0>)
True True True True
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
tensor(4.0811, device='cuda:0', grad_fn=<AddBackward0>)
True True True True
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=


and the output with use_amp=False is -
tensor(4.9359, device='cuda:0', grad_fn=<AddBackward0>)
False False False False
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
tensor(4.8451, device='cuda:0', grad_fn=<AddBackward0>)
False False False False
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
tensor(4.7606, device='cuda:0', grad_fn=<AddBackward0>)
False False False False
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
tensor(4.6795, device='cuda:0', grad_fn=<AddBackward0>)
False False False False
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
tensor(4.6010, device='cuda:0', grad_fn=<AddBackward0>)
False False False False
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
tensor(4.5246, device='cuda:0', grad_fn=<AddBackward0>)
False False False False
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
tensor(4.4501, device='cuda:0', grad_fn=<AddBackward0>)
False False False False
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
tensor(4.3773, device='cuda:0', grad_fn=<AddBackward0>)
False False False False
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
tensor(4.3061, device='cuda:0', grad_fn=<AddBackward0>)
False False False False
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
tensor(4.2365, device='cuda:0', grad_fn=<AddBackward0>)
False False False False
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Process finished with exit code 0


Now irrespective of whatever the loss might be, same, negative or for that matter even nan weights should not be the same. I am even using clone so as to create a copy and compare those. And this behavior is not consistent in the sense that keep rerunning the code above and sometimes the weights are equal and sometimes they are not(when in fp16 mode) . This should not be happening rigth?