I would recommend to scale down the problem a bit and try to overfit a small data snippet (e.g. just 10 samples) to make sure your training code doesn’t have any obvious errors.
If that’s not working out of the box, you could play around with some hyperparameters.
Let us know, if you get stuck.