Hi folks, I am struggling with getting repeatable result for a DETR model I am running with pytorch.
I have defined the following function:
import os
from numpy import random
import numpy as np
def seed_torch(seed=42):
random.seed(seed)
os.environ[‘PYTHONHASHSEED’] = str(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed) # if you are using multi-GPU.
torch.backends.cudnn.benchmark = True
torch.backends.cudnn.deterministic = True
Please note that I have also tried this function with cudnn_benchmark set to false.
This is the folder structure for DETR and the py files contained within it.

I have tried using the seed function in each file separately to see if anyone of them would make my results reproducible and this did not work.
I then tried using the seed function in all files in each folder separately and this did not work.
I tried using the seed function in the first three folders and the first epoch was repeatable but after the first epoch the numbers were different.
The numbers I am talking about look like this:

But I am comparing losses for now.
In the end I copied the seed function to every py file and then used the function in every class and definition in the py files and the results are still not reproducible.
Does anybody have any idea how to resolve this?
I am running my code in the Anaconda terminal.