Hello @ojturner,
From your code



 ojturner:

 huggingface_model = HuggingFaceModel(
    role="-",
    model_data="-",
    entry_point="-",
    source_dir="-",
    code_location="-",
    transformers_version='4.12.3',
    pytorch_version='1.9.1',
    py_version='py38',
)



it looks like you are using a custom inference.py script is that correct? could you provide? Have you tested the latency and overhead using the “zero-code” deployment, without providing inference.py.
Could you also share more information about which model/model-architecture/task you are using?