Thank you for your reply!
From a macro point of view, because I set the result to be printed every 100 steps, I just saw that on my 1660ti laptop with cuda10.2 and torch1.7.0, this output is very fast (because textcnn is not a very complicated network), and about 2 lines of output per second can be jumped out. But on 3090, I need to wait for nearly 10 seconds to see a result output.
Also thank you for pointing out the asynchronization problem. I am sorry but I am not so sure how to use torch.utils.benchmark in multiple lines code… But I revised my timing method as
    for step, (x_batch, y_batch) in enumerate(train_loader):
        torch.cuda.synchronize()
        t1=time.clock()
        x_batch = x_batch.to(device)
        y_batch = y_batch.to(device)
        torch.cuda.synchronize()
        t2=time.clock()
        print('transfer data needs %s ms' % ((t2 - t1) * 1000))

        torch.cuda.synchronize()
        t3=time.clock()
        output = model(x_batch)
        output = output.squeeze()
        loss = loss_func(output, y_batch)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        torch.cuda.synchronize()
        t4=time.clock()
        print('training step uses %s ms' % ((t4 - t3) * 1000))

As I usually train the network for 7 epochs, to make the network ‘warm-up’, I just record the time result of the last steps of the last epoch.
The situation indeed changed! This time on the 3090 machine, the data loading time needs about 0.038ms, while training steps take around 106ms. While my laptop still needs around 0.07ms to transfer data and 4ms to do the train step…
Then the problem becomes why the GPU training is slow on 3090, so I used another method torch.autograd.profiler.profile() to see what happens:
    for step, (x_batch, y_batch) in enumerate(train_loader):
        torch.cuda.synchronize()
        t1=time.clock()
        with torch.autograd.profiler.profile(use_cuda=True) as prof:
            x_batch = x_batch.to(device)
            y_batch = y_batch.to(device)
        torch.cuda.synchronize()
        t2=time.clock()
        print('transfer data needs %s ms' % ((t2 - t1) * 1000))
        print(prof)

And I also use this method on training part mentioned above.
Due to it that the result is too long, I will add two pictures. It seems that the aten::cudnn_convolution operation took the most time? Which seems to have some similarities with the problem you mentioned. (We both used a convolutional structure, I will post my model definition on the next reply.)
Finally, by the way, though I do not quite understand the linked binary… as it seems that the problem happens on cudnn, I just remember that I installed the cudnn library through package-manager-ubuntu-install, maybe this can also be a factor causing this problem? I will try to copy cudnn to /usr/local/cuda-11.2/ rather than /usr/lib/x86_64-linux-gnu/ later.
cost_of_data_transfer1816×349 44.5 KB
cost_of_training_steps2121×3273 479 KB