I am experiencing a 3-4x slower speed when applying the following simple code on newer hardware:
import torch
import time

device = torch.device('cuda:0')

class NNClass(torch.nn.Module):
    def __init__(self):
        super(NNClass, self).__init__()
        self.fc1 = torch.nn.Linear(3, 128)
        self.fclast = torch.nn.Linear(128, 3)

    def forward(self, inp):
        out = self.fc1(inp)
        return self.fclast(out)

theNN = NNClass().to(device)
lossfunc = torch.nn.MSELoss()
inp = torch.zeros(size=(64, 3), device=device)
labels = torch.zeros(size=(64, 3), device=device)

tic = time.perf_counter()
for idx in range(10000000):
    Fout = theNN(inp)
    loss = lossfunc(Fout, labels)
    loss.backward()

    if idx % 1000 == 0:
        print(f'idx: {idx}')
    if idx == int(3e4):
        break

toc = time.perf_counter()
print(f'wallclock: {toc - tic}')

The inferior hardware is a laptop (Lenovo X1 Carbon) connected to an eGPU containing a 1080ti, and the superior hardware is a desktop with an i9-10940x processor (with an ASUS Pro WS X299 Sage II motherboard) with three GPUs – two 2080ti’s and one 1080ti. Both systems are running Ubuntu 18.04 with nvidia driver 460, and both have PyTorch installed using Anaconda with the instructed install command:
conda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch

Basically, when I run the exact same code on both systems, the one with superior hardware is about 3-4x slower when training with the GPU (I trained with one GPU for the superior hardware). The inferior hardware runs in about 12 seconds, while the superior hardware finishes in 46 seconds. As a sanity check, when I train both systems with the CPU only, the superior hardware wins as expected.
Any help is much appreciated!