ReLU will have the value to be zero when the input is below zero. This “flat line” zero will make gradient descent algorithm difficult, because the gradient of a “flat line” is zero. LeakReLU is introduced to resolve this problem, cause the output of a LeakReLU will be a nearly flat line, but not exactly flat.