You could scale down the problem and try to (perfectly) overfit a small data snippet, e.g. just 10 samples from your dataset.
If your model is not able to overfit these samples, you might need to further fine tune the hyperparameters.
Let me know, if this still doesn’t work, as there might then be subtle code bugs, which I’m missing at the moment.