Hi Deep!



 dllacer:

if we use ReLU as activation function after each layer, we are removing negative values during training, so the network will never be able to correctly predict our targets.


No, this way of looking at it isn’t correct.
The reason is that the weights in your layers can be negative and
the biases are not constrained.  So a negative input value can be
multiplied by a negative weight or have a positive bias added to it,
and therefore become positive, before it gets to the ReLU activation.
Best.
K. Frank