HI,
How do you check memory usage? If you use nvidia-smi, you should know that by default, the numbering from nvidia smi is not the same as the one for other programs.
You can set export CUDA_DEVICE_ORDER=PCI_BUS_ID to make all applications use the same numbering as nvidia-smi.
Also to avoid using GPU you should not, you can use CUDA_VISIBlE_DEVICES=1 to prevent your application from seeing any other gpu that the number 1 (this ordering is the other programs one).