The memory usage given in nvidia-smi will give you the reserved memory in PyTorch (allocated + cached) as well as the CUDA context (and all other processes).
Setting cudnn.benchmark=True will run different cudnn kernels for each new input shape and select the fastest one. This behavior might use more memory in the initial iteration and the cached memory will thus increase. In case a profiled algorithm uses a workspace, which would cause an out of memory issue, the algo is skipped.
To check the allocated memory, you can use torch.cuda.memory_allocated().