How are you measuring the memory usage?
Note that PyTorch uses a custom memory allocation, which caches the device memory.
nvidia-smi will this give you the memory usage of the CUDA context, the allocated as well as the cached memory.
Since you have to reduce the batch size, it seems that the memory might indeed to growing.
Since you are observing this behavior in a lot of use cases, could you post a simple code snippet to reproduce this behavior, please?