Hello,
I am experiencing a strange issue with replicating the same results on two different platforms.
I trained a neural network for medical imaging segmentation on a supercomputer using an Nvidia Volta V100 GPU, torch version 1.4.0, CUDA 10.1, evaluating its results using Dice scores.
When a reviewer asked for additional measures of the network’s accuracy, I downloaded the code on my machine and generated its outputs again, obtaining an entirely different, higher Dice score result (average of ~0.95 vs ~0.82).
I can easily replicate this result on both platforms: running exactly the same code (by copying and pasting the full directory), loading the same weights, I still obtain two different values. In my local environment I am also using version 1.4.0 and CUDA 10.1, of course using a different GPU (GeForce RTX 2080 Ti). I also made sure to test this by having the same version of numpy in both environments (1.18.1).
Edit: I am also running the same python version: 3.7.6, with the only difference that I am running it in a conda environment on my local machine. The only difference could be the GCC version: 7.3.0 on the V100 machine, 7.5.0 on mine.
Do you have any idea where this difference could be coming from?