We have been deploying a BERT model to a SageMaker endpoint using the g4dn.xlarge instance. The deploy is managed with terraform and we use the following image:
{account}.dkr.ecr.{region}.amazonaws.com/huggingface-pytorch-inference:1.7-transformers4.6-gpu-py36-cu110-ubuntu18.04

We’re really happy with the model latency, which is about 0.05 seconds for the typical request.
Now we were experimenting with using the more recent image:
{account}.dkr.ecr.{region}.amazonaws.com/huggingface-pytorch-inference:1.9.1-transformers4.12.3-gpu-py38-cu111-ubuntu20.04 

and found that model latency dropped by a factor of about 10, everything else being equal. I noticed in the sagemaker sdk docs that it says the supported versions of pytorch, transformers and python respectively are 1.7.1, 4.6.1 and 3.6 respectively.
So my question is more of a curiosity - why does the model latency drop on this most recent image? Is there some issue using the GPU here?
Thanks,
Owen