I assume the little memory will be allocated on GPU0.
If that’s the case, you can run your script using:
CUDA_VISIBLE_DEVICES=1 python script.py args

to hide GPU0 in your script. GPU1 will then be remapped to GPU0, so that you would have to use 'cuda:0'or just 'cuda' in your script.
Which PyTorch version are you using btw? I thought this issue was solved recently.
I haven’t seen this issue before, but apparently it’s related to some multiprocessing functions in Python.