Could you share the script you are using to save the model?

Hi this is what I am using. I am new to this, so not sure if this is the right way.

from transformers import T5EncoderModel
import torch

pretrained_model_name = "hkunlp/instructor-large"
output_path = "/tmp/traced_t5.pt"

model = T5EncoderModel.from_pretrained(pretrained_model_name, torchscript=True)
model.eval()

dummy_input = model.dummy_inputs["input_ids"]
traced_model = torch.jit.trace(model, dummy_input)
torch.jit.save(traced_model, output_path)

reload

loaded_model = torch.jit.load(output_path)
loaded_model.eval()

r = loaded_model(dummy_input)

This code runs. But when I try loading the model to opensearch, I get this error:

"forward() Expected a value of type 'Tensor' for argument 'input_ids' but instead found type 'Dict[str, Tensor]'.\nPosition: 1\nDeclaration: forward(torch.transformers.models.t5.modeling_t5.T5EncoderModel self, Tensor input_ids) -> ((Tensor))"}","is_async":true}

Do I need to give it a different dummy input when tracing? I also tried passing model.dummy_inputs as example_kwarg_inputs but that did not work as well.

Hi, to fix the problem, you may try the following:

Instead of passing the entire dictionary (model.dummy_inputs), pass only the input_ids tensor.

dummy_input = model.dummy_inputs["input_ids"]
traced_model = torch.jit.trace(model, (dummy_input,)) 

When you reload the model and want to test it, ensure you're passing just the tensor, not the dictionary:

r = loaded_model(dummy_input)