One reason could be that different models will benefit differently from quantization. If your model’s weights aren’t huge then perhaps you were already close to compute bound in the float version.
Another reason could be that we haven’t spent a ton of time optimizing our OpenCL kernels. Perhaps they aren’t using the best instructions possible. This is something we would love for improvement on. You can find the kernels in the .cl files located here 2.