One possible reason is that it takes time to send the data to the GPU. When a calculation is very quick as in this case there are some other small time costs that could add up. If you want to check whether what I am saying is correct try and do a QR decomposition on a matrix that takes a few minutes to run on a cpu and then try it on the GPU.