Solved by albanD in post #2 
              
              
                Yes the update step for GD and SGD are the same  and so from the optimizer point of view they are both the same. 
If you want to the S to be meaningful, you can see it as Sub Gradient Descent  But here again if you had real gradients, the update would be the same. 
It is not possible to save gradien…