If you have doubt about your training, I would suggest to scale down the problem and e.g. use only a small subset of your data (e.g. 10 samples) and try to overfit your model.
If the loss cannot converge towards zero, your code might have some bugs or the model is just not suitable for your use case.