No, the experiment on fp64 is to check if errors are similar for that implementation as well but it looks like that’s not the case.
The main problem here is not moving from fp64 to fp32 and back, but the fact that the cpu implementation appear to not be equal to the expected formula for fp32 implementations.
On CUDA F.linear(x, W, B) == x @ W.T + B
On CPU F.linear(x, W,B) != x @ W.T + B
Similar problems occur for other layers like for other layers like conv1d, conv1dt as well.
So the question here is that why is the CPU implementation not equal to expected formula?
I have a suspicion that the implementation for fp32 is different than that being used for fp64 for ex. XNNPack only supports fp32, and that is causing errors that accumulate.