We propose a new generic type of stochastic neurons, called q-neurons, that considers activation functions based on Jackson's q-derivatives, with stochastic parameters q. Our generalization of neural network architectures with q-neurons is shown to be both scalable and very easy to implement. We demonstrate experimentally consistently improved performances over state-of-the-art standard activation functions, both on training and testing loss functions.
############ Updated Review #################
I have read the author(s)' rebuttal. My decision stays unchanged. In my opinion, this first step is not significant enough, and the presentation is clearly below the acceptance threshold for ICLR. Additionally, the author(s) did not update their submission to reflect the changes. I thereby recommending rejection to this submission. 
##########################################
This work proposes to replace the regular deterministic activation functions used in artificial neural nets with stochastic variants. In particular, the author(s) considered the q-derivatives of standard activation functions. 
The author(s) claimed that ``By Proposition 2, the p-derivative of the q-activation g_q(x) agrees with the original activation function f.'' I have trouble understanding this. I assume by original activation function the author(s) meant f(x), then how can Eqn (4) agree with f(x)?
At the bottom of pp. 3, the author(s) wrote: ``q-neuron ... combines stochasticity and some second-order information in an easy-to-compute way.'' I definitely can not agree with this point. Basically, q-neuron is the ``derivative'' of the original activation function, so there is no surprise that its derivative links to the second derivative of f(x). I can always use the high order derivative of some function as activation and claim now we are combining even higher order information into the neural network, but does that help? I don't think so. 
It really annoys me to see that four out of the eight pages are occupied by gigantic figures, which should be placed in supplementary material in my opinion. A simple table could do the job equally well in the text. We are not interested in nitty-gritty details on how the training evolves. Let alone the datasets tested are all small-scale image classification tasks. At least the author(s) should diversify their test beds (e.g., NLP tasks and ImageNet scale experiments) and model architectures (e.g., RNN, ResNet). 
What's also missing from their experiments is a fair comparison with the real counterparts. I do not see comparisons with dropouts, and to more direct activation function randomization schemes (additive noise to regular activation functions). 
To summarize, I can not approve this paper as it falls well below the acceptance level of an ICLR. In its current form, it's more like a sketchy note rather than a serious academic paper. I would encourage the authors to significantly enrich the content of this writing before considering resubmitting to another venue.