The large variance (1.15 vs 1.83) seems to suggest there are some sort of contention. I wonder if that is caused by the data loading. What if we use CUDA event elapsed_time to measure dog(img_tensor)? Note that time. monotonic() does not guarantee to give the correct measurements as there could still be CUDA ops pending in stream.
Or is it possible to get a self-contained example that we can investigate locally? E.g., using torch.rand to create random inputs instead loading from screenshot.png