Hi,
The multi-gpu performance will depend a lot on how much work the gpu has to do. were you fully using one gpu already? How much data do you transfer to the gpus when forwarding?
Multi-gpu comes at the cost of model synchronization, if your model is small, this cost will be larger than the time to perform the forward pass itself.