I&#x27;m not sure what you mean by &#x27;valid&#x27; or &#x27;matters&#x27; or the difference between them though.<p>I&#x27;m asking if it is more likely that the 0.5% difference is due to the change in the implementation than normal variation in the original implementation.<p>I don&#x27;t know how you can say that this is the case based only on the information that it&#x27;s a geometric mean. You need to know the variation in the reference implementation, and you don&#x27;t.<p>The facts that he measured an increase of performance of 0.5% and he used a geometric mean are not enough information to say that it&#x27;s statistically significant.