Dropout has been one of standard approaches to train deep neural networks, and it is known to regularize large models to avoid overfitting. The effect of dropout has been explained by avoiding co-adaptation. In this paper, however, we propose a new explanation of why dropout works and propose a new technique to design better activation functions. First, we show that dropout can be explained as an optimization technique to push the input towards the saturation area of nonlinear activation function by accelerating gradient information flowing even in the saturation area in backpropagation. Based on this explanation, we propose a new technique for activation functions, gradient acceleration in activation function (GAAF), that accelerates gradients to flow even in the saturation area. Then, input to the activation function can climb onto the saturation area which makes the network more robust because the model converges on a flat region. Experiment results support our explanation of dropout and confirm that the proposed GAAF technique improves performances with expected properties.
This paper offers the argument that dropout works not due to preventing coadaptation, but because it gives more gradient, especially in the saturated region. However, previous works have already characterized how dropout modifies the activation function, and also the gradient in a more precise way than what is proposed in this paper. 
## Co-adaptation
co-adaptation does not seem to mean correlation among the unit activations. It is not too surprising units need more redundancy with dropout, since a highly useful feature might not always be present, but thus need to be replicated elsewhere.
Section 8 of this paper gives a definition of co-adaptation,
based on if the loss is reduced or increased based on a simultaneous change in units.
https://arxiv.org/abs/1412.4736
And this work, https://arxiv.org/abs/1602.04484, reached a conclusion similar to yours
that for some notion of coadaptation, dropout might increase it.
## Gradient acceleration
It does not seem reasonable to measure "gradient information flow" simply as the norm of the gradient, which is sensitive to scales, and it is not clear if the authors accounted for scaling factor of dropout in Table 2.
The proposed resolution, to add this discontinuous step function in (7) with floor is a very interesting idea backed by good experimental results. However, I think the main effect is in adding noise, since the gradient with respect to this function is not meaningful. The main effect is optimizing with respect to the base function, but adding noise when computing the outputs. Previous work have also looked at how dropout noise modifies the effective activation function (and thus its gradient). This work, http://proceedings.mlr.press/v28/wang13a.html, give a more precise characterization instead of treating the effect as adding a function with constant gradient multiplied by an envelop. In fact, the actual gradient with dropout does involve the envelope by chain rule, but the rest is not actually constant as in GAAF.