Hello people, I’m doing a very classical image classification exercise using pytorch, with a CNN network with Drop Out, and I’m using cuda to use my GPU, I ran my learning loop and I get very good execution time ~30s/epoch, everything is fine, but after 2 hours when I rerun the code I see that this same learning loop goes to ~10min/epoch, I went back to the base code where there is no modification (where I had ~30s/epoch times just 2 hours before), but I still have ~10min/epoch, I reinstalled my NVIDIA drivers with CUDA, and the torch, torchvision and cudatoolkit libraries, but nothing solved the problem, I asked a friend to try who has the same PC as me and he gets the ~30s/epoch well.
For your information, I am on Windows 10 (without using Anaconda). And to check the use of my GPU, I look directly in the task manager of my Windows 10, which displays the percentage of GPU usage of each program.
I put you some print to show you that CUDA works well.
import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

print(device)
print(torch.cuda.get_device_name())
print(torch.__version__)
print(torch.version.cuda)
x = torch.randn(1).cuda()
print(x)

output :
cuda
NVIDIA GeForce GTX 1060 3GB
1.10.2+cu113
11.3
tensor([-0.6228], device='cuda:0')

and my very simple learning loop :
learning_rate = 0.01
momentum = 0.5
batch_size_train = 40
batch_size_test = 500

data = loadImgs(batch_size_train=batch_size_train, batch_size_test=batch_size_test)
model = Net().to(device)
optimizer = torch.optim.SGD(net.parameters(), lr=learning_rate, momentum=momentum)

num_epochs = 20

for j in range(num_epochs):
    batch_enum = enumerate(data.loader_train)
    i_count = 0
    iterations = data.num_train_samples // data.batch_size_train
    loss_list = []
    for batch_idx, (dt, targets) in tqdm(batch_enum):
        i_count = i_count+1
        outputs = model(dt.to(device))
        loss = F.cross_entropy(outputs, targets.to(device))
        loss_list.append(loss.item())
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if i_count == iterations:
            break

    torch.save(net.state_dict(), './data/model_TP'+str(time.time())+'.pt')