I would recommend to try to overfit a small data sample (e.g. just 10 samples) and make sure your model is able to do so.
If that doesn’t work out of the box, you could play around with some hyperparameters.
Once this is done, you could (carefully) scale up the problem again.