There is a similar issue here 3. A lot of stuff might be happening.

Are you sure your labels are correct?
Did you try other learning rates? smaller? bigger?
Did you try other architectures? You might be suffering from under-fitting.
Did you try other losses?
Did you normalize your training data?