It is a TorchScript model? I found that the model is really slow during first ~10 iterations due to some optimisation passes. For my sequence model this was unacceptable as I was passing multiple different lengths, where each new length seemed to be causing optimisation to run.
Can you try:
with torch.jit.optimized_execution(False):
  # call your model here.