I would guess this is the “nature” of the sigmoid activation, as the gradients will have the highest value at f(x=0) = 0.5, which would push the output towards the limits.
Usually you would use this activation for a binary or multi-label classification use case.
If you are dealing with a regression, I would try to remove the output activation and check, if it’ll work.