Based on the size, the CUDA context seems to be initialized on GPU0 (and I thought we got rid of this, but cannot find the issue on GitHub).
Anyway, if you only want to use a specific device, you execute your script via
CUDA_VISIBLE_DEVICES=1,2 python script.py

to mask all other devices.
Note that internally in your script, the GPUs will be remapped starting at index0.