I think the problem is that the average of gradient is small in every batch, because you feed the whole dataset into the model and optimizer at one time.
I am not sure how large your dataset is. When the batch size is too large and the data is too noisy, the final gradient will be averaged to near zero. Then, the model will converge at an extremely slow rate.
Try to feed your data one by one to the model and optimizer. See what happens.