Hi,
I am facing an issue with multiple forward passes using torchscript on GPU. It works fine on the first pass but throws an error on the next. However, it works perfectly on CPU.
model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=False).cuda()
model.eval()
scripted_model = torch.jit.script(model)

torch.jit.save(scripted_model, "maskrcnn_torchscript.pth")

scripted_model = torch.jit.load('maskrcnn_torchscript.pth').eval()
x = cv2.imread('image.jpg')
x_tensor = torch.as_tensor(x.astype("float32").transpose(2, 0, 1)).cuda()

with torch.no_grad():
    count = 10
    while count > 0:
        t0 = time()
        out = scripted_model([x_tensor])
        t1 = time()
        print("Time = {}, FPS = {}".format(t1 - t0, 1/(t1 -t0)))

        count -= 1

The first pass works fine and on the next pass I get the following error
code/__torch__/torchvision/models/detection/mask_rcnn.py:95: UserWarning: RCNN always returns a (Losses, Detections) tuple in scripting
Time = 1.6492173671722412, FPS = 0.6063482109181317
Traceback (most recent call last):
  File "simple_torchscript.py", line 24, in <module>
    out = scripted_model([x_tensor])
  File "/path/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
RuntimeError:
Arguments for call are not valid.
The following variants are available:

  aten::size.int(Tensor self, int dim) -> (int):
  Expected a value of type 'Tensor' for argument 'self' but instead found type 'List[Tensor]'.
  Empty lists default to List[Tensor]. Add a variable annotation to the assignment to create an empty list of another type (torch.jit.annotate(List[T, []]) where T is the type of elements in the list for Python 2)

  aten::size.Dimname(Tensor self, str dim) -> (int):
  Expected a value of type 'Tensor' for argument 'self' but instead found type 'List[Tensor]'.
  Empty lists default to List[Tensor]. Add a variable annotation to the assignment to create an empty list of another type (torch.jit.annotate(List[T, []]) where T is the type of elements in the list for Python 2)

  aten::size(Tensor self) -> (int[]):
  Expected a value of type 'Tensor' for argument 'self' but instead found type 'List[Tensor]'.
  Empty lists default to List[Tensor]. Add a variable annotation to the assignment to create an empty list of another type (torch.jit.annotate(List[T, []]) where T is the type of elements in the list for Python 2)

The original call is:

However, if I run it on CPU, it works perfectly.
code/__torch__/torchvision/models/detection/mask_rcnn.py:95: UserWarning: RCNN always returns a (Losses, Detections) tuple in scripting
Time = 2.616811990737915, FPS = 0.3821443816137551
Time = 3.022855758666992, FPS = 0.3308130059242312
Time = 2.174025297164917, FPS = 0.4599762483463605
Time = 2.174373149871826, FPS = 0.45990266208858743
Time = 2.098878860473633, FPS = 0.47644483863844345
Time = 2.3181285858154297, FPS = 0.43138245484697213
Time = 2.0808985233306885, FPS = 0.4805616366142636
Time = 2.05479097366333, FPS = 0.48666750672803294
Time = 1.95816969871521, FPS = 0.5106809694053165
Time = 2.1022541522979736, FPS = 0.47567987862309613

Here’s the details of the versions used.
----------------------------------------------------------------------------------------------------------------
python               3.6.12 |Anaconda, Inc.| (default, Sep  8 2020, 23:10:56) [GCC 7.3.0]
Numpy                1.19.2
PyTorch              1.7.0
torchvision          0.8.1
GPU 0,1,2,3          Quadro RTX 6000
CUDA_HOME            /usr/local/cuda-10.2
NVCC                 Cuda compilation tools, release 10.2, V10.2.89
cv2                  4.4.0
-------------------  --------------------------------------------------------------------
PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - CUDA Runtime 10.2
  - CuDNN 7.6.5