So until very recently, PyTorch used two CUDA RNGs the MTGP32 and the Philox 4x32 10 3.
My impression is that what you are getting different state values for the former - I must admit I’m ignorant of the specifics why that is (they’re generated using curandMakeMTGP32KernelState from the cuRAND API).
The happy news is that with the PyTorch nightlies, you can profit from Syed Ahmed’s awesome work on the random generation in PyTorch 20. Since last week or so, master and the nightlies use only the Philox generator for CUDA and the problem should be gone. (Note that the only used state now is seed and offset from the Philox generator, so 2 64-bit numbers in the last 8 bytes of the state.)
Best regards
Thomas