Hi, I got the same problems.
I trained same model, same input, same batch size, same epochs on 1 GPU and 8 GPUs. for the parallell, I use loss.mean().backward
But on the contrast, my parallel result is better than just on one GPU by a sizable margin.
I don’t understand which represents the real performance. Could anyone explain it?