Answers.Body: <p>The sigmoid activation is defined as follows: <span class="math-container">$\sigma(x)=\frac{1}{1+e^{-x}}$</span>, this means that for <span class="math-container">$x\to\infty$</span> the activation saturates to one whereas for <span class="math-container">$x\to-\infty$</span> it saturates to zero.</p>
  <p>In your example you're trying to visualize the activation per neuron, in feature (or input) space. What I see is that if you sum the shaded blue regions (the ones with activation close to 1) you are able to identify the positive class quite well. Basically, each neuron is learning to threshold one side of the triangle where the negative class resides.</p>
  <p>The output layer can sum these activation and rescale such that a further sigmoid, returns values close to <span class="math-container">$1$</span> for points that fall outside the triangle.</p>
  <blockquote>
  <p>I can conclude higher activations from layer 1 give an output class of 0.</p>
  </blockquote>
  <p>I don't see how you can say so. You shouldn't reason in feature space, but rather also account for the value of the weight that multiply each feature. Maybe, you want to say that, for a given neuron, a lower value of the feature corresponds to a higher activation but that depends on the sign of the weight the feature is multiplied with.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-16T14:15:18.660Z
Answers.LastActivityDate: 2023-07-16T14:15:18.660Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>The example is trying to predict wether coffe is well roasted or badly. 1 is good roasted and 0 is bad. The architecture is:<a href="https://i.sstatic.net/mO1yl.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/mO1yl.png" alt="architecture" /></a></p>
  <p>Now I try to visualize the model.
  <a href="https://i.sstatic.net/tH3qU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/tH3qU.png" alt="visualization" /></a>
  Unit 1 has higher values when the duration of roasting is too little. Unit 2 has higher values for bad combinations of temperature and time. The blue-hatched regions show the layer outputted higher activations for the respective unit.</p>
  <p>I used a threshold of 0.5
  So I can conclude higher activations from layer 1 give an output class of 0. But as I used the sigmoid activation function I thought that large values give an output class of 1. And very small values give a 0, due to the S-shape.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-14T17:27:22.903Z
Comments.Text: Can you please highlight the question? Now sure what to answer.
Comments.UserDisplayName: Chinmay
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-14T18:37:11.027Z
Comments.Text: Maybe you have negative weights in your output layer. What are the 3 weights that compute a2 from a1 ?
Comments.UserDisplayName: Lelouch
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-14T19:56:10.203Z
Comments.Text: if it's a 2 layer network or more, you cannot reason anymore like this
Comments.UserDisplayName: Alberto
Tags: <neural-networks><classification>
Title: Effect of large activations of hidden layers
UserDisplayName: Jacky02
ThreadText: # Effect of large activations of hidden layers
  
  (Asked by: username_0 on 2023-07-14)
  
  The example is trying to predict wether coffe is well roasted or badly. 1 is good roasted and 0 is bad. The architecture is:[![architecture](https://i.sstatic.net/mO1yl.png)](https://i.sstatic.net/mO1yl.png)
  
  Now I try to visualize the model.
  [![visualization](https://i.sstatic.net/tH3qU.png)](https://i.sstatic.net/tH3qU.png)
  Unit 1 has higher values when the duration of roasting is too little. Unit 2 has higher values for bad combinations of temperature and time. The blue-hatched regions show the layer outputted higher activations for the respective unit.
  
  I used a threshold of 0.5
  So I can conclude higher activations from layer 1 give an output class of 0. But as I used the sigmoid activation function I thought that large values give an output class of 1. And very small values give a 0, due to the S-shape.
  
  **Tags:** neural-networks, classification
  
  > **Comments:**
  > **username_1**: Can you please highlight the question? Now sure what to answer.  
  > **username_2**: Maybe you have negative weights in your output layer. What are the 3 weights that compute a2 from a1 ?  
  > **username_3**: if it's a 2 layer network or more, you cannot reason anymore like this  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2023-07-16. Score: 0
  
  The sigmoid activation is defined as follows: $\sigma(x)=\frac{1}{1+e^{-x}}$, this means that for $x\to\infty$ the activation saturates to one whereas for $x\to-\infty$ it saturates to zero.
  
  In your example you're trying to visualize the activation per neuron, in feature (or input) space. What I see is that if you sum the shaded blue regions (the ones with activation close to 1) you are able to identify the positive class quite well. Basically, each neuron is learning to threshold one side of the triangle where the negative class resides.
  
  The output layer can sum these activation and rescale such that a further sigmoid, returns values close to $1$ for points that fall outside the triangle.
  
  > I can conclude higher activations from layer 1 give an output class of 0.
  
  I don't see how you can say so. You shouldn't reason in feature space, but rather also account for the value of the weight that multiply each feature. Maybe, you want to say that, for a given neuron, a lower value of the feature corresponds to a higher activation but that depends on the sign of the weight the feature is multiplied with.
  
  
  ---
  
  