The problem was that I was implementing dropout on a linear function which is after the activation function.
According to the paper on Variational Dropout I’ve mentioned before, one should implement the dropout mask before the non-linear activation (sigmoid / tanh in my case…)