There may be indeed a bottleneck from using one process instead of one-process per GPU, but seeing no speedup suggests that it indeed something else that is bottlenecking the GPUs… Generally it is preferable to get close to 100% utilization on a single GPU before moving to multiple GPUs.