CUDA operations are called asynchronously, so you should synchronize the code before starting and stopping the timer using torch.cuda.synchronize().
In your current code snippet logits = logits.detach().cpu().numpy() will create a synchronization point, so that your code will wait at this line of code for all preceding operations to finish (which also might include the forward pass of your models) while the timer was already started.