I have four gpus, I make one program to load four models and inference each model with shared data in each single gpus, then merge each model’s results. For example:
models = [model1.to('cuda:0'), model2.to('cuda:1'), model3.to('cuda:2')]
data = get_data()
results = []
for index, model in enumerate(models):
    data = data.to('cuda:%d'%index)
    r = model(data)
    result.append(r)
final_result = fusion_result(result)

But right now, each model can only get data from a single process, can I get data and share to multi-model and do asynchronous inference by multiprocessing? Many thanks.