As a quick test, I would recommend to try to overfit a small data sample (e.g. just 10 samples) and make sure your model is able to overfit it.
If that’s not the case, you would need to play around with some hyperparameters (e.g. learning rate).