You could start by overfitting a small dataset (e.g. just 10 samples) by playing around with the hyperparameters (optimizer, learning rate etc.).
If that doesn’t help, I would suggest to simplify the model.
Once your can come up with a model and hyperparameters, which overfit this small dataset, you could scale up the use case again by using more data.