Hi everyone!
I am fine tuning the following model 2  using the script provided. However, even if I follow the reproducibility guidelines 1, I obtain different losses across experiments with the same seed. I used PyTorch Lightning seed_everything function and also tried doing it manually for each random source, as follows:
import torch
import os
import numpy as np
import random as r
os.environ['PYTHONHASHSEED'] = str(1)
np.random.seed(1)
r.seed(1)
torch.manual_seed(0)
torch.cuda.manual_seed(0)
torch.cuda.manual_seed_all(0)
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True

Finally I tried freezing all model and only training the fully connected layer at the end, and in this case it is reproducible, meaning that the source of randomness is inside the backbone. I tried with torchvision’s resnet 18 3D, and it does not happen, so I am quite certain that the problem is with the CUDAExtension the authors provided for the shift module. Is there a way to fix the seed on the CUDA module?
Thank you!