Indeed. Given that all of them become nan at the same time, seems that it’s something else. Does the nan always occur at the same iteration? Also, what loss function are you using? It might be useful to also print out the output and target too, as that gives a nan for the loss value!