Solved by Dwight_Foster in post #2 
              
              
                Use case #2 is the best. That is the one recommended by the pytorch docs. You could use 3 if you set the stepsize really high but usually doing it in minibatches will lower the learning rate too fast.