It is first important to localize the error. You could try one of two things.

First try different learning rates, while simultaneously turning of all regularization.
Try using a standard loss function like the MSE (for regression) or the Cross Entropy (if classes are present). See if these loss fucntions decrease for a particular learning rate. If these losses do not decrease, it may indicate some underlying problem with the data or the way it was pre-processed.