Thanks! One last thing:
I find it strange that there is this problem with distributed training because apparently when you put a model on multiple gpus in pytorch, all the distributed stuff dissapears once it returns its prediction according to: https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html#create-model-and-dataparallel 1 .
Therefore, once I return y_pred in the infer_dict of the training engine, nothing is distributed anymore?