Where is the batch size used in your code?
As you said, the batch size should be unrelated to the copying of parameters.
Since you are using non_blocking=True, make sure to call torch.cuda.synchronize() before starting and stopping the timer to get valid results.