Ok, I played with it a bit more and can confirm both points.
If any GPUs are in MIG mode in the system, CUDA does not see the non-MIG GPUs.
(venv_a100) [sergey_mkrtchyan@a100-demo cformers]$ nvidia-smi -L
*[Omitting some GPUs for brevity]*
GPU 4: A100-PCIE-40GB (UUID: GPU-4dc4cf75-1fd9-6d80-74ce-c26a52cecd47)
MIG 1g.5gb Device 0: (UUID: MIG-GPU-4dc4cf75-1fd9-6d80-74ce-c26a52cecd47/7/0)
MIG 1g.5gb Device 1: (UUID: MIG-GPU-4dc4cf75-1fd9-6d80-74ce-c26a52cecd47/8/0)
MIG 1g.5gb Device 2: (UUID: MIG-GPU-4dc4cf75-1fd9-6d80-74ce-c26a52cecd47/9/0)
MIG 1g.5gb Device 3: (UUID: MIG-GPU-4dc4cf75-1fd9-6d80-74ce-c26a52cecd47/10/0)
MIG 1g.5gb Device 4: (UUID: MIG-GPU-4dc4cf75-1fd9-6d80-74ce-c26a52cecd47/11/0)
MIG 1g.5gb Device 5: (UUID: MIG-GPU-4dc4cf75-1fd9-6d80-74ce-c26a52cecd47/12/0)
MIG 1g.5gb Device 6: (UUID: MIG-GPU-4dc4cf75-1fd9-6d80-74ce-c26a52cecd47/13/0)
GPU 5: A100-PCIE-40GB (UUID: GPU-909ef3d1-5b61-d36a-0f03-ae7b85464540)
MIG 7g.40gb Device 0: (UUID: MIG-GPU-909ef3d1-5b61-d36a-0f03-ae7b85464540/0/0)
GPU 6: A100-PCIE-40GB (UUID: GPU-67260e8f-5838-5b68-c651-1077b44f562f)
GPU 7: A100-PCIE-40GB (UUID: GPU-3761d9cd-7737-fa3e-a8c6-00265e242f04)
(venv_a100) [sergey_mkrtchyan@a100-demo cformers]$
(venv_a100) [sergey_mkrtchyan@a100-demo cformers]$ export CUDA_VISIBLE_DEVICES='6,7'
(venv_a100) [sergey_mkrtchyan@a100-demo cformers]$
(venv_a100) [sergey_mkrtchyan@a100-demo cformers]$ python
Python 3.8.1 (default, Jan 21 2020, 07:30:43)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> import os
>>> os.getenv('CUDA_VISIBLE_DEVICES')
'6,7'
>>> torch.cuda.device_count()
1
>>> torch.cuda.current_device()
/mnt/nlu/users/sergey_mkrtchyan/workspace/cformers/venv_a100/lib/python3.8/site-packages/torch/cuda/__init__.py:104: UserWarning:
A100-PCIE-40GB MIG 1g.5gb with CUDA capability sm_80 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70 sm_75.
If you want to use the A100-PCIE-40GB MIG 1g.5gb GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/
warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
0
>>>

Notice how it grabs 1g.5gb MIG instance.
In MIG mode you are free to grab any instances in separate processes, but cannot grab two at a time.
This makes MIG a little hard to swallow I would say.
Thanks a lot for the help, @ptrblck !