Re MMS_DEFAULT_WORKERS_PER_MODEL=1 and latency, unfortunately I don’t have any data on that.
I have an example for an inference.py script here: text-summarisation-project/inference.py at main · marshmellow77/text-summarisation-project · GitHub 3, I hope this is useful.
Now, re debugging - agreed that it would be ideal to locally debug the deployed inference code. I know that Sagemaker offers “local mode”, which (in theory) should allow you to do exactly that: Use the Amazon SageMaker local mode to train on your notebook instance | AWS Machine Learning Blog 1
So far the pain wasn’t high enough to try it myself, I always got by with print statements in the inference code so far. But in case you want to give the local mode a try I’d be curious to learn whether it works well for you