Misconception about ReLu I have already gone through the post and this post, but they didn't clear my doubt. Let us say if I have a deep neural network like (having more layers about 50):

Now, my question is:
If I'm using an activation function as the ReLU, the gradient will be 1 for all values of x>0 and 0 for for all values of x<0. So, where do the notion of vanishing gradients occur for a ReLU? I thought a ReLu was known for solving the vanishing gradient problem.

ReLUs don't suffer from vanishing gradients, but they have their own issues.
The main issue for ReLU is the dead-ReLU phenomenon, i.e. a neuron whose activation becomes zero for all samples due to ReLU. This neuron is irrecoverable, because gradients through it are always zero.
Leaky ReLU and other activation functions were created to deal with this issue.
