Do you mind providing some more information about your model and task?
I think it may be hard to diagnose the issue without it. My initial thought is that some processing task each time is maxing out one of the CPU threads and is bottlenecking the model but I can’t say without knowing the model and task and how it’s implemented. I’m not sure if someone else has any other thoughts?