title: Mean Replacement Pruning  
abstract: Pruning units in a deep network can help speed up inference and training as well as reduce the size of the model. We show that bias propagation is a pruning technique which consistently outperforms the common approach of merely removing units,  regardless of the architecture and the dataset.  We also show how a simple adaptation to an existing scoring function allows us to select the best units to prune.  Finally,  we show that the units selected by the best performing scoring functions are somewhat consistent over the course of training, implying the dead parts of the network appear during the stages of training.
authors: Utku Evci Nicolas Le Roux Pablo Castro Leon Bottou
hIndex: Leon Bottou 69