Recently I found that in my case Adam already has negative value in state['exp_avg_sq'] so this is explains why do I need only optimizer state loaded to reproduce the issue because
exp_avg_sq.mul_(beta2).addcmul_(grad, grad, value=1 - beta2) - which does not help
and
denom = (exp_avg_sq.sqrt() / math.sqrt(bias_correction2)).add_(group['eps'])

Which is immediately produces NaN value
See adam.py file
Why did Adam accept invalid values in its state is a good question… So now I need to track its state to discover moment when it is get negative value in  state['exp_avg_sq'] …