when training a neural network, if we pass our input through ReLU, then it makes negative values zero, but this does not lead to a normal or symmetric distribution, for example,
import torch, seaborn
input = torch.randn(1, 3, 28, 28)
rlu = nn.ReLU()
seaborn.distplot(input.view(-1).detach().numpy())

gives

after applying ReLU,
seaborn.distplot(rlu(input).view(-1).detach().numpy())

this distribution becomes,

do we not want to preserve symmetry of this distribution during training?
same thing happens when we use,
g = nn.CELU()
seaborn.distplot(g(((input))).view(-1))


h = nn.ELU()
seaborn.distplot(h(((input))).view(-1))


i = nn.SELU()
seaborn.distplot(i(((input))).view(-1))