@aplassard  Note that nvidia-smi is not impacted by the CUDA_VISIBLE_DEVICES env variable.
@PengZhenghao from a state where it works, could you open two new shells.

Check that nvidia-smi shows all the gpus in both.
Run CUDA_VISIBLE_DEVICES=0,1 on one shell.
Check that nvidia-smi shows all the gpus in both still. Is that the case?
Run export CUDA_VISIBLE_DEVICES=0,1 on one shell.
Check that nvidia-smi shows all the gpus in both still. Is that still the case?
In each shell, run python then inside import torch and print(torch.cuda.device_count()). One should return 2 (the shell that had the export command) and the other 8. Is that the case?