Based on the loss you’ve posted, it seems the model is initially learning but diverges eventually.
You could try to play around with some hyperparameters (such as the learning rate or trying another optimizer).
If that doesn’t help, you could try to overfit a small data sample first (again by changing some hyperparameters) and make sure your model is able to do so. Once this is done, you could carefully scale up the problem again.