Alright, it was caused by not using torch.cuda.synchronize().
I had to call this command before calling the timer and before calculating the final duration. Only then I started getting consistent timing.