Sam-gege:

does pytorch calculates everything in float64 but saves its parameters using float32?


No, PyTorch uses float32 by default for the parameter storage and all calculations.
If you are using mixed-precision training via torch.cuda.amp then some operations will be performed with float16 inputs/outputs, but the computation will still be performed in float32.
The difference might come from the limited floating point precision and a different order of operations between PyTorch and numpy.