As long as it’s decreasing, it’s a good sign. We can probably speed it up with an adaptive optimizer like Adam, but in the debug phase it’s kind of useless to tune the optimizer.
Is you network architecture taken from some working example or did you just come up with it?