I don’t know how you defined your model but you should also use DDP to maximally parallelize the models with multiple GPUs & use DistributedSampler with multiple processes.


Do you mean using DDP for inference for this case?