Solved by ptrblck in post #2 
              
              
                I would guess this is the “nature” of the sigmoid activation, as the gradients will have the highest value at f(x=0) = 0.5, which would push the output towards the limits. 
Usually you would use this activation for a binary or multi-label classification use case. 
If you are dealing with a regressio…