Solved by ptrblck in post #2 
              
              
                One difference would be the loss scale, since you are using the mean reduction in the first approach and a sum in the latter. Also, the latter trains for 1000 epochs with the entire dataset (all 2000 samples), while the former approach trains a single epoch with a single sample per batch in each ite…