Or you can try to check every one second if GPU is busy (I actually don’t know how to do it, I expect it to be super easy, but I never did it). If not, you run inference on the accumulated batch, if yes you wait another second and check again. This way the worst case scenario is the same 7 seconds, but you do not make people wait unnecessarily when load is light and the wait wasn’t needed.