Hello guys!
For research issues we rented a new workstation (workstation 1) with three NVIDIA RTX A5000 GPU (ubuntu 18.04LTS OS, CUDA 11.1 and spyder) because we need to train a much bigger model with 3D training samples. To test the performances of the model, I decreased the size of each training sample to 50 by 50 by 50 and compare the performances of the old workstation (workstation 2) with three NVIDIA RTX 2080Ti GPU (CUDA 10.2, ubuntu 16.04LTS and spyder) and the rented one using the same python script. The results shows that in the training process each epoch ran for 0.21 minutes on workstation 2 but it ran for 1.38 minutes on workstation 1. I wanted to find out the issues because there is no reason that A5000 GPU runs slower than NVIDIA RTX 2080Ti. The performances are listed as follows:
Workstation 1:
GPU:three NVIDIA RTX A5000 GPU
Parallelization technique: 
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
if str(device)=="cuda:0":
    print('Use Cuda GPU!')

if torch.cuda.device_count() > 1:
    print("Let's use", torch.cuda.device_count(), "GPUs!")
    net = nn.DataParallel(net)
optimizer = optim.Adam(net.parameters(), lr=0.001)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=20, shuffle=False, num_workers=0)
Let's name three GPU as GPU 0, 1 and 2
Training time for each epoch:
1. only GPU 0 without DataParallel: 1.38 min (ubuntu OS unaffected)
2. only GPU 1 with DataParallel: 1.41 min (OS become slow, moving other window will display frame by frame)
3. GPU 1 and GPU 2 working in parallel : 6 min (OS become slow, moving other window will display frame by frame)
4. GPU 0,1,2 working in parallel: 3.8 min (OS become slow, moving other window will display frame by frame)


Workstation 2:
GPU:three NVIDIA RTX 2080Ti GPU
Parallelization technique: 
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
if str(device)=="cuda:0":
    print('Use Cuda GPU!')

if torch.cuda.device_count() > 1:
    print("Let's use", torch.cuda.device_count(), "GPUs!")
    net = nn.DataParallel(net)
optimizer = optim.Adam(net.parameters(), lr=0.001)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=20, shuffle=False, num_workers=0)
Let's name three GPU as GPU 0, 1 and 2
Training time for each epoch:
1. GPU 0,1,2 working in parallel: 0.21 min (OS become slow, but not severe as Workstation 1)

Could you please provide any solutions so that we can improve the performances on A5000 GPU? Thanks a lot!