For others who might be interested: I tested running multiple jobs on all 8 GPUs in a single instance (P2 instance on AWS, K80 GPU). When running a single job nvidia-smi shows about 2% usage per GPU. Running a second model spikes the usage of all GPUs to 100% and the jobs choke --literally stop running. So nope.
Running with CUDA_VISIBLE_DEVICES works as expected, but I have a couple of questions:


I notice that torch.cuda.current_device() is always assigned to 0, regardless of which GPUs are made visible in the environment. Not sure if this is a problem but I would have assumed that if only GPUs 4,5,6,7 are visible then current_device should be one of these?


With the exact same model settings as when using 8 GPUs, with 2 GPUs only the job is using around 5-40% of each GPU. Expected. But an epoch takes around 7 minutes with 8 GPUs but only 2 minutes with 2. What gives?