Solved by tom in post #4 
              
              
                I’m not sure whether I understand this correctly, but so basically, once you have inf, you are most likely in trouble: 
When you get inf bias, one might expect an inf gradient to also propagate to the earlier operations (because for a + b, both a and b get the same gradients as the sum). 
But once …