I agree with @lewtun i’d first diagnose what’s causing the timeout. You can add logs to your inference functions (model loading and inference) and check the ModelLoadingWaitTime SageMaker endpoint metric in CloudWatch to check what steps take time. If model loading is the issue, according to the SageMaker doc 2 it may speed things up if you use a d type instance, that have local NVMe SSDs instead of using virtual EBS drives for storage. Also, try using bigger instances (in AWS, network bandwidth is generally proportional to instance size).
If using regular SageMaker endpoints is enough (and not SageMaker MME), you can take a look at that demo that deploys the bigger bert-base-cased on SageMaker GitHub - aws-samples/amazon-sagemaker-bert-classify-pytorch: This sample show you how to train BERT on Amazon Sagemaker using Spot instances 5