Okay thanks,
I purposefully raised it that high for hyperopt, but I guess I conclude that that is way to high.
Does such problems also effect the performance of batchnorm layers. I have had also issue where my was lr=0.1. And using model.eval() also led to exploding validation loss?