There’s a number of avenues that you could use to reduce inference time:

Scale your deployment vertically or horizontally.
Move to a smaller model.
Improve your preprocessing/postprocessing efficiency.