It’s a bit hard to speculate what might be the root cause of this issue.
I would generally recommend to try to overfit a small dataset (e.g. just 10 samples) by playing around with some hyperparamters and make sure your current model as well as training setup is able to do so.
Once this is done you could scale up the use case again.