Are you running out of memory? Maybe the memory is just cached and looks like it’s used in nvidia-smi.
You can find more information about the memory management here 13.
If you want to release the cached memory to the OS, you could call torch.cuda.empty_cache().