I don’t know what kind of data you are using, but your model might just not have enough capacity.
Try to overfit using a smaller sample size, e.g. 10 samples and see how the training loss behaves.
If that looks good, you could add some more samples and see it the model is still able to learn the data.