if you compare the gradients of 1.2 and 1.3, you will find 1.3’s gradient is 1.2’s gradient multiplied by the batch_size, which in this case is 64. Another way to look at this would be that 1.3 is equivalent to training using 1.2 but with the learning rate multiplied by the batch_size = 64. This would be a pretty large effective learning rate and might be causing problems with the optimization. if you are doing 1.3, i suggest dividing the learning rate by 64.