Solved by ptrblck in post #4 
              
              
                No, that shouldn’t be the case as the compute and accumulate dtype would still be performed in FP32 even if FP16 inputs are passed.
If depends on the range of your values and if they are representable. Theoretically it would be possible, e.g. via:

x = torch.tensor(2**(-149)).cuda()
print(x)
> tens…