Hi !!
I am taking a pre trained alexnet model and using it to classify an image. However a strange thing is happening. The time it takes to classify the image on the GPU is coming out to be 800-1000 milliseconds. However when I run this in a loop , then time of image inferencing is almost about 2 milliseconds for each iteration after the first one. The first on still takes the same time. I am using the same image for all. I am doing it something lie this :
for i in range(100):
       pre_trained.cpu()
       model_transfer_start = time.time()
       pre_trained.cuda(0)
       torch.cuda.synchronize()
       model_transfer_end = time.time()
       ans = pre_trained(img_variable)
       infer_end = time.time()

img_variable is the preprocessed image Variable.
The results are something like this : (All times are in milliseconds)
Iteration 1: 872.12313
Iteration 2: 2.14568
Iteration 3: 2.43156
Iteration 4: 2.00987
Iteration 5: 2.01218
Iteration 6: 3.01121
and so on…
What am i doing wrong ? And if nothing then why is this such big gap ?