It seems your model or training procedure might have a bug.
The best way of making sure your code works is to try to overfit your model on a small sample (e.g. just one single sample).
If your model cannot overfit the single sample, something in the architecture or training might be wrong.
Could you try to do that and report the results?