UPDATE
I was able to make my results reproducible by using the following code:
def set_seed(seed):
    torch.manual_seed(seed)
    np.random.seed(seed)
    random.seed(seed) 
    # for cuda
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.enabled = False

I call the set_seed function with every run I do. Tbh, I don’t fully understand why I cannot just set the seed once, but it works