Hi, I am trying to reproduce a NN in Python, using PyTorch. I set in my code np.random.seed(3)  and torch.manual_seed(3) (the numpy seed being the same as the one used for the NN without Pytorch, and the torch seed, being whatever value).
I know the results are different with different initial seed because the optimizer will start in a different point, but the differences are huge. If I choose torch.manual_seed(1) the train and test accuracy are 65% and 34%, respectively. If I choose  torch.manual_seed(3), they are instead 97% and 68%. My datasets consist on a training set of only 209 images and 50 testing images. I am using SGD instead of gradient descent. Is there any reason to think that gradient descent would be less affected on the initial conditions than the SGD? (different seeds giving similar accuracies)
I would expect strong differences if I would be working with a huge dataset as CIFAR-10 for example, but not with a small dataset as the one described. Am I wrong in the logic I follow?