I’m not answering your question, but I noticed that you have pre_trained.cpu() in your code. I assume that “pre_trained” is the name of your neural net? If so, I cannot understand why you cast it to cpu() before casting it back into GPU, and, worse, why do you have all this inside a loop? If you want to perform inference on GPU it would make more sense to not cast your model to CPU…
I think that the first 800-1000 ms is due to moving all necessary stuff onto your GPU, setting up buffers etc. (but I’m no specialist).
Also I assume that img_vairable is .cuda()