Of what magnitude is the difference between the prediction and the target?
Did you try varying the learning rate/learning rate decay?
It might be possible that the learning rate is simply to large which causes oscillations around the minimum and prevents the optimizer from finding the global minimum.
You could imagine of a valley where you want to reach the ground. If you start at one side and your stepsize is too large you will only be able to get somewhere near the bottom but you will never reach the ground.