I am sorry. But today this phenomena happens again.
After tons of testing, I find the key point:

Using device_id=0,1,2,3 in DataParallel

Using nohup to run your python script
Using a bash script to set CUDA_VISIBLE_DEVICES and run python

I don’t have time to find why this happen. So just run my code under those odd requirements. Codes are  provided in case it may help somebody or staff to figure the reason.
In my bash script:
#!/usr/bin/env bash
CUDA_VISIBLE_DEVICES=0,1,2,3 nohup python main.py --fuse-type=cnn --batch-size=256 --model=cnn --epochs=50 --lr=5e-4 2>&1 &
echo 'Start training!'

In my code:
    if args.model == 'cnn':
        model = CNNBase(4, feature_length, num_class)
    elif args.model == 'mlp':
        model = MLPBase(feature_length, num_class)
    model = model.cuda()
    model = torch.nn.DataParallel(model, device_ids=[0,1,2,3])

UPDATE:
I find a more clean way to deal with this problem…
That is declare device_id in DataParallel without setting the environment variable.
When I use device_id=[0,1,2,3] and let torch.cuda.device_count() return 8. The program still work in 4 gpus and in nvidia-smi 8 gpus are visible.