I am trying to optimize a model which has output range [1, -1] so, I know for the last layer I can not use ReLU as it always gives positive output. But, I was wondering that , can using ReLU in intermediate layers affect the model output as it will make each activation output greater than 1. Thus choosing other activation function such as TanH will be more suitable case here. Or, may be Leaky ReLU. I guess, In case of Leaky ReLU, setting the slope is a crucial hyperparameter. I tried all three activation functions, TanH optimizes over time, ReLU model give nan output after few epoch, Leaky ReLU some time work some time gives output as nan.  Can someone have their thoughts on this topic.