I’ve noticed a strange behaviour of my network that I can’t explain: I trained a network, evaluated it on a separate test set, got an accuracy score, called the evaluation function again and got a different value.
A bit more details:
I use Jupyter Notebook in google colab. I load the data as follows:
full_dataset = datasets.ImageFolder(image_path, transform=transform)
train_size = int(train_split * len(full_dataset))
test_size = len(full_dataset) - train_size
train_dataset, test_dataset = random_split(full_dataset, [train_size, test_size])
trainloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
testloader = DataLoader(test_dataset, batch_size=batch_size)
classes = train_dataset.dataset.classes

then I define a CNN (pretrained resnet18), train it and evaluate it using this code (the output is binary):
def evaluate_model(mod):
  mod.eval()
  correct = 0
  total = 0
  with torch.no_grad():
    for data in testloader:
      imgs, labels = data[0].to(device), data[1].to(device)
      out = mod(imgs)
      preds = (out > 0.5).float()
      l = labels.unsqueeze(1).float()
      total += l.size(0)
      correct += (preds == l).sum().item()
  return ((correct/total)*100)

This function gives me significantly different values if I call it multiple times without changing anything in-between. How is that possible?
I used every option I know to fix the randomness to see if that helps, but the result is still the same.
seed = 123
import os
os.environ['PYTHONHASHSEED'] = str(seed)
torch.manual_seed(seed)
np.random.seed(seed)
random.seed(seed)
torch.cuda.manual_seed(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False