After looking up the code of Pytorch’s SGD:
Pytorch’s SGD 97
it seems (excuse me in advance if my assertion is wrong) that the SGD is not a real SGD. Indeed, the way gradients are accumulated and more especially the order wherein they are accumulated is up to the user. Thereupon where is the randomness or stochastic character of this gradient descent?
According to Wikipedia:
SGD 41
the examples must be randomly shuffled in the training set after being used to compute the gradient. However here this stage is carried out by the user when he accumulated the gradient with the backward.