I would recommend to set the environment variables in the terminal before running the script, as setting it via os.environ might have side effects.
E.g. if you are loading a library, which calls to CUDA and grabs all available GPUs, setting 'CUDA_VISIBLE_DEVICES' won’t have any effect anymore, as it apparently the case in your code.
If you really want to set it in your code, make sure os.environ is called first in your script.