When you measure the runtime of forward, did you use cudaStreamSynchronize(...) (which is equivalent to torch.cuda.synchronize() in Python)? This can affect runtime measurement because CUDA is by default asynchronous.