Since CUDA operations are asynchronous, the host to device copy via .cuda() could create a synchronization point and thus accumulate the timing from the actual forward and backward pass.
If you would like to profile the code manually, you could add manual synchronization points via torch.cuda.synchronize().