Hi,
I have been trying to debug to find out why it takes too long to train a CNN regressor with the following training code:
img = img.cuda()
img = Variable(img)

output = model(img)
target = target.type(torch.FloatTensor).unsqueeze(0).cuda()
target = Variable(target)

loss = criterion(output, target)

losses.update(loss.item(), img.size(0))

optimizer.zero_grad()
loss.backward()
optimizer.step()

The problem
This is a blocking issue for my research since increases the training time in unacceptable amounts.
(former training time : 15h-18h, current: 45h-52h)
I have used perf_counter to measure the time needed for each line of above code and realized that backpropagation takes ~10x of the time needed for forward pass. I know that backprop is expected to take longer than forward pass, but I believe this difference is a bit too much.
The tests
Thinking that it might be an infrastructural problem, I also have ran my code at colab environment. The result was more or less the same. Here are the timings of training steps for first 7 images from the first epoch of both trainings.
colab              local                   
forward  backward  forward  backward                                      
0.077s   0.818s    0.071s   0.699s         
0.025s   0.238s    0.095s   1.043s         
0.071s   0.751s    0.066s   0.751s         
0.046s   0.486s    0.063s   0.685s         
0.073s   0.778s    0.060s   0.619s         
0.039s   0.385s    0.015s   0.139s         
0.077s   0.818s    0.019s   0.176s         

I appreaciate any help on finding or rectifying the flaw.