Brando_Miranda:

so why is my loss at a scale of -16 if that’s much lower than what it should handle as far as I understand? In a different place I have e-7 …so have I hit zero or not?


1e-6 is not the absolute minimal value before the value is rounded to zero as explained e.g. here 6.
As you can see in the Precision limitation on decimal values section, the fixed interval between “small integer values” is approx. 1e-7, which is why this can be used as the minimal step size between these values.
The interpretation is of course depending on your use case and e.g. in the ML domain absolute errors of ~1e-6 are commonly accepted.
Here is a small example using the minimal denormal values as well as larger values with a fixed interval of 1:
torch.set_printoptions(precision=30)

a = torch.tensor(2**(-149)) # smalles denormal number
print(a)

b = torch.tensor(2**(-150)) # not representable
print(b)

x = torch.tensor(1.)
y = x + torch.tensor(2**(-23)) # addition representable
print(y)
print(x - y)

z = x + torch.tensor(2**(-24)) # not representable
print(z)

c = torch.tensor(2**23) # step size is 1
d = c + torch.tensor(0.5) # rounded
print(d - c)
e = c + torch.tensor(1.) # representable
print(e - c)