What did you mean here?

Comes from the code sample with gradcheck above. Playing with the value of k, you get from 2 to 4 digits of precision in the computed gradient.

So you might suspect that the difference between the automatic differentiation and the numerical approximation with torch.autograd.gradcheck() comes from a numerical precision issue.

Yes I think this is where it comes from.
The gradients from finite difference are not more correct than the ones computed with the autograd. Just that they do different approximation and so give a different result at the end. Both should be good enough to use in a stochastic optimization procedure.