Hi,
Thanks for your quick reply and explaination! I agree with you that the extra memory occupation is due to the cuda context being initialized on GPU 0. However, I was wondering if there was a solution that allowed me to specify which GPU to initialize the cuda context.
The usecase of torch.cuda.empty_cache() is that I need to call a subprocess to achieve an exhaustive test after 10 epoch training. Releasing GPU cached memory allows this subprocess to utilize them and avoid OOM.
Using environment variables is very convenient. As for CUDA_VISIBLE_DEVICES, I’m sure that CUDA_VISIBLE_DEVICES can save me from such a circumstance for now. However, it will modify the environment variable, and I may need other processes to perform some deeplearning tasks on other GPUs with potentially different framework, like TensorFlow. As a result, I’d like to find a way than can constrain the GPU utility locally, trying not to mess up with the environment variables.