Yeah I know that you usually check things like finite differences approximation against autograd in float64 however I thought that since I am computing analytical expression against analytical expression for a very small network it should not matter that much. But I am no expert on single precision calculus and I was wondering if things like the order of the multiply/sum mattered and stuff like that and how easy it was to make things match perfectly.