You are taking sums of the errors, and matmul has large sums, too. In general, the approach you take, ie verifying that going from fp32 to fp64 reduces the error by a 1e7ish factor, is a good quick check for being confident that you are seeing numerical accuracy issues.