Hello everyone,
I’ve been trying to train my model and achieve reproducible results, but unfortunately, I’m not able to.
I’ve taken this code from an old thread, but that still didn’t make it work:
def set_random_seed(seed_value, use_cuda=True):
np.random.seed(seed_value)  # cpu vars
torch.manual_seed(seed_value)  # cpu  vars
random.seed(seed_value)  # Python
os.environ[‘PYTHONHASHSEED’] = str(seed_value) # Python hash buildin
if use_cuda:
torch.cuda.manual_seed(seed_value)
torch.cuda.manual_seed_all(seed_value)  # gpu vars
torch.backends.cudnn.deterministic = True  # needed
torch.backends.cudnn.benchmark = False
I also set all num_workers=0 in the data loaders to avoid a well-known problem there.
I’ve noticed a strange thing that happens after several training iterations (2 similar runs):
image1146×580 27.6 KB
there is a slight change in the calculation that leads to totally different results.
Did anyone face this problem? is it even fixable?
Thanks!