I guess you should also include some of your training code to help troubleshoot. The code you’ve provided here looks ok.
Given that it happens after a few epochs I guess the gradient is either vanishing or exploding. Either one could be caused by a learning rate issue. For exploding gradients you could try gradient clipping.