Just some thoughts. They might or might not be helpful.
Set a parameter which would control frequency of inference calls in seconds. Let incoming requests accumulate until time for the next scheduled inference call comes. When it does, group all what came in until that time into one batch and run inference on that batch. If nothing is in queue, then just wait until next scheduled inference time. Naively one might expect if you, let’s say, set this frequency to 4 seconds, then it gives GPU enough time to process the previous request and stand ready for your next call. This way everybody gets a response in no longer than 7 seconds. Reality could be different, but this is something to play with. Hope it helps.