Unfortunately, I cannot download the file (the site claims it’s downloading but nothing happens).
I also cannot reproduce the issue using the max. values in float16:
x = torch.ones(5665, 48, device='cuda', dtype=torch.float16) * 65504
print(x)
with torch.cuda.amp.autocast():
    out = x - x.mean()
print(torch.isnan(out).any())
> tensor(False, device='cuda:0')