I would recommend to scale down your problem a bit and try to overfit a small fraction of your dataset (e.g. just 10 samples) by playing around with the hyper-parameters.
Once your model can learn this dataset, you could try to scale it up again by using more data.