In case this still doesn’t meet your latency requirements, my next suggestion would be to quantize the model’s weights: Quantization — PyTorch 1.7.1 documentation 5
It basically amounts to running this line of code
model_int8 = torch.quantization.quantize_dynamic(
    model,              # the original model
    {torch.nn.Linear},  # a set of layers to dynamically quantize
    dtype=torch.qint8)  # the target dtype for quantized weights

but you should do some sanity checks to make sure the resulting accuracy hasn’t degraded too much (in which case you can try 16-bit precision).
If you want to take it further, you can try serialising the model with ONNX (although this can be a bit finnicky): Exporting transformers models — transformers 4.3.0 documentation 5