If you are using the GPU you would have to synchronize the code before starting and stopping the timer.
Otherwise the time will be accumulated in a blocking operation.
Could this be the case for the different timings you are seeing?