Try to overfit a small data sample (e.g. just 10 samples) to check, if the overall training routine doesn’t have any bugs.
I’m not completely sure, how your data loading pipeline works, so this might also be a check for this part of the code.
Once your model overfits the small sample, try to scale it up.