What is loss function?
Loss function is very vital in learning.
Loss function should make intuitive sense of your end goal for learning.
Secondly, What is the scale for your loss?
check for how loss is decreasing. If it starts from lower value and decreases slowly, then probably your network is not yet learning.
I would suggest following-

Validate loss function and verify if it is designed to do what you expect
Plot the loss and validate if it is decreasing and how fast