I am experiencing a similar issue. I load an image to the gpu, apply a model, then transfer the result (which is a 4x61x114 tensor) to the cpu. The first steps take a few milliseconds, the last one around half a minute. Except for the model’s loading, code is as follows:
img = io.imread(path + f).astype(np.float32) / 255
start = time.time()
x = torch.from_numpy(img.transpose(2,0,1)).unsqueeze(0).cuda()
with torch.no_grad():
    y = model(x).squeeze().cpu()
print(time.time() - start)

If I comment out the .cpu() step it takes some milliseconds, otherwise several seconds. What is going on? What can I do about it?