You are most likely creating the CUDA context on the default device and/or accidentally allocating additional memory on it.
Mask the device you would like to use via CUDA_VISIBLE_DEVICES=1 python script.py args and use cuda:0 in the script to select the masked device.