I cannot find any obvious errors in your code and would recommend to try to use a small data sample and overfit it.
If your model isn’t able to overfit e.g. 10 samples, then I might be missing a bug in the code or the overall training pipeline or model might not be suitable for the task.