@kehuantiantang can you please clarify the nature of the data, is it pinned memory on CPU or it is on GPU? It might be more efficient to put it on one of the GPU devices and share it with others via IPC instead of transferring it from device to device using .to() call.
Also it seems that you are looking for solution to evenly distribute load between GPUs as one of the models are much faster than others. In this case I recommend to try outer loop which will rotate models  between GPUs (if memory allows) and avoid fusion_result calls as long as possible as it is looks like your synchronization point.