ZimoNitrome:

That’s true. Do you know if this is the case for ReLU aswell?


LeakyRelu/ReLU doesn’t need to know anything about the shape of the input, so it’s good ;).
Keep at least the Leaky ReLu in place, otherwise your linear layers stacked together simply become a basic perceptron (i.e a single linear layer).