Could you lower the probabilities so that the next parameter update will only take place if the loss is lower and would be discarded if the loss increases?
If the loss still increases, there seem to be a bug in your code somewhere.