I trained a 3D unet model by Pytorch 1.1.0 Distributed Data Parallel. The test result is good. But after I update Pytorch to V1.7.0. Same code different result. Anyone can help me with that? Is it a syncbatchnorm problem or what?
Even if I finetune the pretrained model(on 1.1.0). Both losses during training and validation increased a lot using v1.7.0.
Below is the training epoch loss of two version.