by default the model should be cached in ~/.cache/huggingface/ (see docs 1) so maybe you can inspect the heroku machine to see if that’s the case?
i still think the best approach would be to wrap you logic in a fastapi / flask app and then deploy that on heroku. this will allow you to separate the loading of the model from generating predictions and will be much faster. of course you can also try out streamlit which is easy to get started with