CUDA operations are asynchronous, so you should synchronize via torch.cuda.synchronize() before starting and stopping the timer.
Most likely, your code waits at the x = x.cuda() calls, while the model is still running in the background, which will accumulate the actual model time into the time to transfer the data.