This suggests your network is not designed properly. I would suggest you simplify your model to a couple of layers to start with and train it with 1 mini batch. Once confirmed the network can reliably converge, you can start adding more layers.