Are you seeing a speedup or slowdown?
I’m a bit confused, since you’ve mentioned both.
Also, how are you measuring the memory usage?
Note that nvidia-smi shows the total allocated memory (CUDA context + cache + allocated memory), so you should use torch.cuda.memory_allocated() instead.