Hi,
thanks a lot for your reply.
You’re right - it seems I simplified the code too much and thought it still captures the issue, but it does not.
So let me try again. I have to add a bit more code to reproduce it: a seq model with 3 conv ops with dilation and padding. I hope the way I time the code is ok (I do a cuda sync before querying the time).
import torch
import torch.nn as nn
import numpy as np
from time import time


model = nn.Sequential(nn.Conv2d(in_channels=3,
                                out_channels=64,
                                kernel_size=3,
                                padding=2,
                                dilation=2),
                      nn.Conv2d(in_channels=64,
                                out_channels=128,
                                kernel_size=3,
                                padding=4,
                                dilation=4),
                      nn.Conv2d(in_channels=128,
                                out_channels=128,
                                kernel_size=3,
                                padding=2,
                                dilation=2))
model.eval()
model.to('cuda')

for i in range(10):
    torch.cuda.synchronize()
    t0 = time()

    # create image (numpy array)
    x = np.zeros([720, 1280, 3], dtype=np.uint8)

    # cut out part of the image
    x = x[1:-1, 1:-1]
    x = x.copy()  # and if I remove this line then it is always fast

    # create batch
    x = x[None].transpose(0, 3, 1, 2)

    # copy at each 2nd iteration
    make_copy = bool(i % 2)
    print('copy' if make_copy else 'no copy')
    if make_copy:
        x = x.copy()

    x = torch.as_tensor(x).to('cuda')
    x = x.float()
    x = x/255-0.5

    print('is_contiguous:', x.is_contiguous())

    with torch.no_grad():
        y = model(x)

    torch.cuda.synchronize()
    t1 = time()

    print('dt:', (t1-t0)*1000)
    print()


This gives me:
no copy
is_contiguous: False
dt: 2819.4923400878906

copy
is_contiguous: True
dt: 78.82118225097656

So there is a huge gap on my GPU between copying and not copying. What is interesting is that if the Torch tensor is contiguous everything runs fast, and otherwise it runs slow (so as_tensor creates a non-contiguous tensor from a non-contiguous numpy array!?). I was not able to reproduce with only 2 conv ops stacked, but starting with 3 and with a large enough number of channels I could reproduce (e.g. with all output channels set to 64, I could not reproduce).
Tested on 2080TI and 3090, and with PyTorch 1.8.1+cu111.
P.S.: changed the title, as it is not the function as_tensor that is slow, but somehow the model inference.