I would recommend to try to overfit a small data sample and make sure your model can successfully predict these samples.
If that’s not the case, there might be a bug in the code I’m missing or the architecture is not suitable for the task.