I did not add any try catch in my 1 file code. I also found that NaN appears in generator weights and it is set by Adam optimizer which is uses NON NaN gradients. Also gradient norm clipping which I used did not help. I found that the only thing I need to reproduce the issue is just optimizer itself I do not need original batch or even generator weights. I will try make minimalistic example to reproduce my issue, it is seems that at some point Adam gets crazy.