Try to scale down the problem and try to overfit a small data sample (e.g. just 10 samples) by playing around with the hyperparameters.
Once your model can overfit it, you could try to scale it up again.