Answers.Body: <p>In general, it's better to not use the sigmoid function in any hidden layer. There are many other great options such as ReLU and ELU. However, if for any reason you have to use a sigmoid-like function, then go with the Tanh function, at least it has ~0 mean.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-28T15:28:57.583Z
Answers.LastActivityDate: 2021-12-21T10:39:22.040Z
Answers.LastEditDate: 2021-12-21T10:39:22.040Z
Answers.UserDisplayName: pedrum
Body: <p>I am using a fully connected neural network that uses a sigmoid activation function. If we feed a big enough input, the sigmoid function will finally become 1 or 0. Is there any solution to avoid this?</p>
  <p>Will this lead to classical sigmoid problems vanishing gradient or exploding gradient?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-11T15:15:55.437Z
Comments.Text: I think I am stuck with the use of sigmoid(since I am looking for a probability ) but I wonder how to increase the precision to 256 what do you mean by that ?
Comments.UserDisplayName: ou2105
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-11T17:01:22.463Z
Comments.Text: The other problem is when using sigmoid in an FNN with more than 1 hidden layer is that backpropagation degrades and weights in the first layer will only be updated by small values that might not have a significant impact. Relu fixes this problem. I agree with @DuttaA use softmax if you want to get a probability.
Comments.UserDisplayName: SandMan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-11T17:43:51.403Z
Comments.Text: Can you elaborate on the task you're intending to solve with your network? As others have already said, ReLUs are the best option in intermediate layers and it's probable that you could do with logit (pre-sigmoid) outputs instead of going through the sigmoid for backprop purposes.
Comments.UserDisplayName: David
Tags: <neural-networks><machine-learning><deep-learning><activation-functions><sigmoid>
Title: Why will the sigmoid function be 1 (and 0), if we use a fully connected layer that produces a big enough positive (or negative, respectively) output?
UserDisplayName: ou2105
ThreadText: # Why will the sigmoid function be 1 (and 0), if we use a fully connected layer that produces a big enough positive (or negative, respectively) output?
  
  (Asked by: username_0 on 2019-09-11)
  
  I am using a fully connected neural network that uses a sigmoid activation function. If we feed a big enough input, the sigmoid function will finally become 1 or 0. Is there any solution to avoid this?
  
  Will this lead to classical sigmoid problems vanishing gradient or exploding gradient?
  
  **Tags:** neural-networks, machine-learning, deep-learning, activation-functions, sigmoid
  > **Comments:**
  > **username_0**: I think I am stuck with the use of sigmoid(since I am looking for a probability ) but I wonder how to increase the precision to 256 what do you mean by that ?  
  > **username_1**: The other problem is when using sigmoid in an FNN with more than 1 hidden layer is that backpropagation degrades and weights in the first layer will only be updated by small values that might not have a significant impact. Relu fixes this problem. I agree with @DuttaA use softmax if you want to get a probability.  
  > **username_2**: Can you elaborate on the task you're intending to solve with your network? As others have already said, ReLUs are the best option in intermediate layers and it's probable that you could do with logit (pre-sigmoid) outputs instead of going through the sigmoid for backprop purposes.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2020-07-28. Score: 1
  
  In general, it's better to not use the sigmoid function in any hidden layer. There are many other great options such as ReLU and ELU. However, if for any reason you have to use a sigmoid-like function, then go with the Tanh function, at least it has ~0 mean.
  
  
  ---
  
  