Based on the high memory usage I assume that you are either not deleting all references to CUDATensors or other processes might be using the GPU memory.
PyTorch itself will create the CUDA context, which won’t be released by calling torch.cuda.empty_cache(), and which will use approx. 600-1000MB of device memory.