Try to overfit a small dataset, e.g. just 10 samples, by playing around with the hyperparamters.
Once your model is able to do so, try to scale up the use case again. If your model is not able to perfectly learn these 10 samples, some other bug might be in the code which we haven’t found yet.