Try it for a longer period of time. Because maybe your network is just learning that slow.
Try adam : 0.005 with learning rate decay
Try SGD : 0.05 with learning rate decay
for lets say 30 epochs. If it doesn’t work, then maybe we gotta debug the problem more.