If the loss is increasing then perhaps even 0.001 is too high for the learning rate. Accuracy is not a great metric to look at to debug the code. Basically any neural network should be able to fully learn the training data if you train it for long enough, otherwise you dont have enough parameters in the network. So you should focus on trying to get the training loss to converge to 0. If the loss isnt decreasing, your model isnt training correctly.