If dropout is going to remove neurons, why are those neurons built? I know that Dropout will remove neurons randomly to reduce over-fitting.
If Dropout is going to remove neurons, why are those neurons built? We could remove those neurons from the architecture.
Why don't we do that?

To add to @frank's answer, the reason using dropout is not the same as training a smaller network is that the neurons that are dropped out are randomly selected each time the weights are updated. So while on each iteration only some of the neurons are used and updated, over the entire training cycle all the neurons are trained. According to Jason Brownlee's A Gentle Introduction to Dropout for Regularizing Deep Neural Networks, dropout can be thought of as training an ensemble of models in parallel.
