The higher peak memory usage might e.g. indicate different behaviors.
E.g. some initial copies are needed, if you are passing expanded tensors, which are needed in a contiguous format.
Also, if you are running your script with cudnn.benchmark the initial iteration will benchmark different cudnn algorithms and select the fastest one, which might also create a memory peak (however, this should now yield an OOM issue).