Try to play around with some hyperparameters, e.g. lower the learning rate.
If that doesn’t help, try to scale down your use case (e.g. just use a tiny subset of your data) and check if your model is able to overfit this dataset.