mrshenli:

Is the DDP process the only process using that GPU? The extra size ~500MB looks like an extra cuda context. Does this behavior still persist if you set CUDA_VISIBLE_DEVICES  env var properly (instead of using torch.cuda.set_device(rank) ) before launching each process?


Yup, I don’t have X or anything else running: 
I don’t set CUDA_VISIBLE_DEVICES but I do call torch.cuda.set_device(rank) before instantiating the model / optimizers, etc (see function in initial post).
Edit: not sure how I would be able to set the ENV var when using mp.spawn such that it doesn’t apply to both processes.
Edit2: didn’t realize children can modify their ENV independently of the parent and other processes: https://stackoverflow.com/questions/24642811/set-env-var-in-python-multiprocessing-process 12