Are you measuring the time for a specific operation or the forward/backward pass?
If so, note that you would need to synchronize the code before starting and stopping the timer via torch.cuda.synchronize(), since CUDA operations are asynchronous.