Thanks for the code!
Note that CUDA operations are asynchronous so you would need synchronize before starting and stopping the timer for the GPU operation via torch.cuda.synchronize().
Currently you might time the kernel launch times, which are higher for a smaller batch size (and thus more launches).
Could you add the synchronization and profile the code again?