Concretely, even though I type CUDA_VISIBLE_DEVICES=0,1,2,3 after I enter the conda environment, without running any python code, this phenomena also happens.
In fact, the main.py does a simple PyTorch based neural network training, with dataloader, dataparallel in it.
More info:


Dataparallel using 20 workers.
Instead of setting the environment variable, using os.environ["CUDA_VISIBLE_DEVICES"] = "0, 1, 2, 3" inside the python code can also cause the problem. Turn off the shell, or kill the running process, or waiting it finish can solve it.

for target in loader: target_var = target.cuda(async=True) is used. Maybe the async=True will cause some problem?

Thanks for your fast and quick reply!