If your model training is diverging, you could use “bad” hyperparameters such as a high learning rate or there might be a bug in your code (such as forgetting to zero out the gradients).
As quick test I would recommend to overfit a small data sample (e.g. just 10 samples) and make sure your model is able to fit them perfectly.