Hi @ptrblck,
thank you for your attention. I agree with you on main differences. I am also surprised about how old version of those core libraries makes my code run faster.
Here is my measurement code. I have summarized the result by discarding insignificant timings (>0.01) when I was reporting them, for the sake of simplicity.
a = time.perf_counter()
img = img.cuda()
img = Variable(img)
target = target.type(torch.FloatTensor).unsqueeze(0).cuda()
target = Variable(target)
torch.cuda.synchronize()
b = time.perf_counter()

print('data conversion {:.3f}s '.format(b - a), end='')

a = time.perf_counter()
output = model(img)
torch.cuda.synchronize()
b = time.perf_counter()

print('model forward pass {:.3f}s '.format(b - a), end='')

a = time.perf_counter()
loss = criterion(output, target)
losses.update(loss.item(), img.size(0))
torch.cuda.synchronize()
b = time.perf_counter()

print('loss {:.3f}s '.format(b - a), end='')

a = time.perf_counter()
optimizer.zero_grad()
torch.cuda.synchronize() 
b = time.perf_counter()

print('zerograd {:.3f}s '.format(b - a), end='')

a = time.perf_counter()
loss.backward()
optimizer.step()
torch.cuda.synchronize() 

b = time.perf_counter()
print('backprop {:.3f}s '.format(b - a), end='')
a = time.perf_counter()
optimizer.step()
torch.cuda.synchronize() 
b = time.perf_counter()

print('step {:.3f}s'.format(b - a))

Here is how the timing looked like originally
local

data conversion 0.001s model forward pass 0.045s loss 0.001s zerograd 0.000s backprop 0.496s step 0.002s
data conversion 0.002s model forward pass 0.067s loss 0.000s zerograd 0.000s backprop 0.762s step 0.003s
data conversion 0.000s model forward pass 0.015s loss 0.000s zerograd 0.000s backprop 0.115s step 0.002s
data conversion 0.001s model forward pass 0.071s loss 0.000s zerograd 0.000s backprop 0.759s step 0.003s
data conversion 0.000s model forward pass 0.015s loss 0.000s zerograd 0.000s backprop 0.133s step 0.003s
data conversion 0.001s model forward pass 0.058s loss 0.000s zerograd 0.000s backprop 0.589s step 0.003s