Like you said in subsequent layers the sign will flip out. But, I case of my model I am using ReLU at all layers except the last one which is TanH. So, I think model will have hard time to learn. Also, for Leaky ReLU the slope is crucial parameter.