I guess my first question is if I am doing it right in theory to begin with?
I am not initializing the weights/biases in any specific way, but I am fixing the random seeds as such:
    random.seed(666)
    np.random.seed(666)
    torch.manual_seed(666)
    torch.cuda.manual_seed_all(666)
    torch.backends.cudnn.deterministic = True

Is there a way I can make sure the weights and biases all get initialized the same way?