If you are executing the forward pass on the GPU, you should add torch.cuda.synchronize before starting and stopping the timer, as CUDA operations are executed asynchronously.
Currently you might time the kernel launch times or some other operations, which create a synchronization point.