Here are the things I would do:

Verify the data pipeline
Use another dataset to check whether the model and the training loop is fine
Use another architecture.


A great post around this would be Andrej Karpathy’s blog 1.