Check the output of your network.  It is probably outputting all zeros after a while.  Unfortunately, that’s more a diagnosis of the problem.  I’m not sure how to solve it.  Perhaps try a lower learning rate.