In this paper, we turn our attention to the interworking between the activation functions and the batch normalization, which is virtually mandatory technique to train deep networks currently. We propose the activation function Displaced Rectifier Linear Unit (DReLU) by conjecturing that extending the identity function of ReLU to the third quadrant enhances compatibility with batch normalization. Moreover, we used statistical tests to compare the impact of using distinct activation functions (ReLU, LReLU, PReLU, ELU, and DReLU) on the learning speed and test accuracy performance of standardized VGG and Residual Networks state-of-the-art models. These convolutional neural networks were trained on CIFAR-100 and CIFAR-10, the most commonly used deep learning computer vision datasets. The results showed DReLU speeded up learning in all models and datasets. Besides, statistical significant performance assessments (p < 0.05) showed DReLU enhanced the test accuracy presented by ReLU in all scenarios. Furthermore, DReLU showed better test accuracy than any other tested activation function in all experiments with one exception, in which case it presented the second best performance. Therefore, this work demonstrates that it is possible to increase performance replacing ReLU by an enhanced activation function.
This paper proposes an activation function, called displaced ReLU (DReLU), to improve the performance of CNNs that use batch normalization. Compared to ReLU, DReLU cut the identity function at a negative value rather than the zero. As a result, the activations outputted by DReLU can have a mean closer to 0 and a variance closer to 1 than the standard ReLU. The DReLU is supposed to remedy the problem of covariate shift better. 
The presentation of the paper is clear. The proposed method shows encouraging results in a controlled setting (i.e., all other units, like dropout, are removed). Statistical tests are performed for many of the experimental results, which is solid.
However, I have some concerns. 
1) As DReLU(x) = max{-\delta, x}, what is the optimal strategy to determine \delta? If it is done by hyperparameter tuning with cross-validation, the training cost may be too high.
2) I believe the control experiments are encouraging, but I do not agree that other techniques like Dropouts are not useful. Using DReLU to improve the state-of-art neural network in an uncontrolled setting is important. The arguments for skipping this experiments are respectful, but not convincing enough.  
3) Batch normalization is popular, especially for the convolutional neural networks. However, its application is not universal, which can limit the use of the proposed DReLU. It is a minor concern, anyway.