This is happening because you’re not passing correct gpu id. If you’re not using docker, do nvidia-smi to see GPU ids and then specify which GPU you want. Like this
CUDA_VISIBLE_DEVICES=0 python3 main.py
FYI, if you are specifying multiple gpu ids, you need to ensure that your model has multi gpu support with nn.dataparallel, if not consider using only one.