Hi! I had exactly the same problem when trying to infer with my torch model parsed into a TensorRT model. @John_Deterious gave me two hints to solve it:

Make sure the input is in range of what your model expects
Make sure your input is in the same device your model is

I was having this:
model_input = torch.rand((1, 3, 416, 416))
y = model(model_input)
model_trt = to_trt(model, model_input)
y_trt = model_trt(model_input)

But my model expects a normalized input in range [0, 1] and the model_trt was in device device='cuda:0' So my changes were basically to change the input tensor to be in the expected range and on the GPU (same device):
model_input = torch.randn((1, 3, 416, 416)).clamp(0, 1).cuda()
y = model(model_input)
model_trt = parse_model_trt(model, model_input)
y_trt = model_trt(model_input)