Hi I’m fairly new to pytorch, plz correct me if I’m wrong.
My question is, does pytorch calculates everything in float64 but saves its parameters using float32? I heard pytorch prefers float32 for its speed and memory efficiency, so I wrote my own autograd using numpy and set its dtype to float32 as well.
However, when compared the result with pytorch, it varies by a bit: (for example, the final loss comparison is: numpy autograd:0.005946858786046505 vs pytorch autograd: 0.005946869496256113).
When I used float64 in numpy, the result became almost identical to pytorch: (numpy autograd: 0.9532327802786481 vs pytorch autograd: 0.9532327802786484)
I can’t seem to find an answer elsewhere on this topic, thanks for any help!