Thanks for the update!
In that case I would recommend to try to overfit a small data sample as a simple test.
If your model is not able to overfit this small data sample (e.g. 10 samples), you might have a bug in the code somewhere, which I might have missed.
Once you can overfit, I would try to scale up the experiment slowly and make sure your model is still learning.