Hey @philschmid happy new year!
Was wondering if you had any more thoughts on this issue?
We found that when we compiled the model for use with one of the inferentia endpoints and wrote our own predict_fn, we didn’t have any latency issues with the latest code version. Presumably then the original latency problem stems from making predictions using the huggingface pipeline object in the latest code version?
Thank you, Owen