How large is your dataset? Could you try to use a small sample (e.g. just 10 samples) and try to overfit it first?
If that doesn’t work at all, you could have a bug in your code somewhere or the model architecture might not be suitable for the data.