What learning rate are you using? Even in my case with learning rate 0.01 loss was decreasing but accuracy was always 0. I changed the learning to 0.005 and I could see improvement in accuracy with training time.