Maybe your hypeparameters or the model architecture are not suitable for the data.
I would recommend to use a small sample of your data (e.g. just 10 samples) and try to overfit it using your code as a sanity check. If your model isn’t able to learn this small data sample, you might have other bugs in your code.