Hi,
You should keep in mind that the cuda api is asynchronous, so to get proper measurement, you should always call torch.cuda.synchronize() just before calling time.time().
The difference in timing here is most certainly that one model use the gpu more and so the queuing of the operations corresponding to the loss is slightly longer.