Answers.Body: <blockquote>
  <p>Now let's say an image containing only black pixels is input. If I understand it well this would cause all neurons in the hidden layer to output 1</p>
  </blockquote>
  <p>Not necessarily. In a neural network, there are weights <span class="math-container">$W$</span>, synapses if you will, connecting layers. These weights are randomly initialised, and can be negative. Furthermore, assuming a fully connected network, each hidden activation receives the activations from <em>all</em> neurons in the previous layer as input. A mixture of randomly negatively and positively scaled inputs, summed and passed through a non-linearity, could lead to either a positive or negative output. Hence why an input image of black pixels (0) will not cause <em>all</em> the hidden neurons to also be black (0) or white (1). This of course counts for all subsequent layers too.</p>
  <blockquote>
  <p>What is the correct way to model an artificial neuron that outputs 0 when 1 is input and outputs 1 when 0 is input?</p>
  </blockquote>
  <p>As @Minh-Long Luu points out, you probably don't want or need a neural network to do this...
  In fact, I'm not even sure if this is possible (assuming training via backprop) because it sounds like you're after a <a href="https://en.wikipedia.org/wiki/Heaviside_step_function" rel="nofollow noreferrer">heaviside step-function</a> for the non-linear activation, which would be non-continuous and therefore non-differentiable, making it impossible to train the NN with stochastic gradient descent.
  Perhaps a flipped <a href="https://en.wikipedia.org/wiki/Sigmoid_function" rel="nofollow noreferrer">sigmoid</a> centered at 0.5 would be more appropriate?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-16T10:40:10.613Z
Answers.Text: I would expect a single sigmoid neuron trained with gradient descent to converge to that Heaviside step, assuming no regularization.
Answers.UserDisplayName: Jaume Oliver Lafont
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-16T10:05:17.400Z
Answers.LastActivityDate: 2023-02-16T10:05:17.400Z
Answers.UserDisplayName: thesofakillers
Answers.Body: <p>This inverter is an interesting initial application of an artificial neuron.</p>
  <p>Yes, you need a negative weight and... no, the sigmoid suffices if you do with a close enough approximation.</p>
  <p>See how an unrestricted optimization could progress from a very inexact approximation to the Heaviside step in the limit, flipped and shifted as Giulio suggests.</p>
  <p><a href="https://i.sstatic.net/4gljW.gif" rel="nofollow noreferrer"><img src="https://i.sstatic.net/4gljW.gif" alt="enter image description here" /></a></p>
  <p>If you want to test other numbers, here is the <a href="https://www.wolframalpha.com/input?i=1%2F%281%2Bexp%28-100%281%2F2-x%29%29%29%3B%201%2F%281%2Bexp%28-10%281%2F2-x%29%29%29%3B%201%2F%281%2Bexp%28-%281%2F2-x%29%29%29" rel="nofollow noreferrer">WolframAlpha link</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-16T10:29:42.817Z
Answers.LastActivityDate: 2023-02-16T10:37:02.293Z
Answers.LastEditDate: 2023-02-16T10:37:02.293Z
Answers.UserDisplayName: Jaume Oliver Lafont
Body: <p>Sorry if that is a dumb question. I just started to learn about machine learning.</p>
  <p>I'm reading this book about neural networks:
  <a href="http://neuralnetworksanddeeplearning.com/chap1.html#a_simple_network_to_classify_handwritten_digits" rel="nofollow noreferrer">http://neuralnetworksanddeeplearning.com/chap1.html#a_simple_network_to_classify_handwritten_digits</a></p>
  <p>It explains how an artificial neural network classifies handwritten digits applying weights and biases to inputs and each input is assigned a value between 0 and 1, with 0 being white pixels and 1 black pixels.</p>
  <p>Now let's say an image containing only black pixels is input. If I understand it well this would cause all neurons in the hidden layer to output 1, which means that all neurons in output layer will output 1. This is not the intended behavior.</p>
  <p>What is the correct way to model an artificial neuron that outputs 0 when 1 is input and outputs 1 when 0 is input? Is this done by assigning a negative weight or using a different activation function?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-16T03:43:30.207Z
Comments.Text: If you just want to flip 0 to 1 and vice versa, I don't think you need a neural network to do that...
Comments.UserDisplayName: Minh-Long Luu
Tags: <machine-learning>
Title: How does a sigmoid neuron output 1 with 0 as input?
UserDisplayName: user3361941
ThreadText: # How does a sigmoid neuron output 1 with 0 as input?
  
  (Asked by: username_0 on 2023-02-15)
  
  Sorry if that is a dumb question. I just started to learn about machine learning.
  
  I'm reading this book about neural networks:
  <http://neuralnetworksanddeeplearning.com/chap1.html#a_simple_network_to_classify_handwritten_digits>
  
  It explains how an artificial neural network classifies handwritten digits applying weights and biases to inputs and each input is assigned a value between 0 and 1, with 0 being white pixels and 1 black pixels.
  
  Now let's say an image containing only black pixels is input. If I understand it well this would cause all neurons in the hidden layer to output 1, which means that all neurons in output layer will output 1. This is not the intended behavior.
  
  What is the correct way to model an artificial neuron that outputs 0 when 1 is input and outputs 1 when 0 is input? Is this done by assigning a negative weight or using a different activation function?
  
  **Tags:** machine-learning
  > **Comments:**
  > **username_1**: If you just want to flip 0 to 1 and vice versa, I don't think you need a neural network to do that...  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-02-16. Score: 3
  
  > Now let's say an image containing only black pixels is input. If I understand it well this would cause all neurons in the hidden layer to output 1
  
  Not necessarily. In a neural network, there are weights $W$, synapses if you will, connecting layers. These weights are randomly initialised, and can be negative. Furthermore, assuming a fully connected network, each hidden activation receives the activations from *all* neurons in the previous layer as input. A mixture of randomly negatively and positively scaled inputs, summed and passed through a non-linearity, could lead to either a positive or negative output. Hence why an input image of black pixels (0) will not cause *all* the hidden neurons to also be black (0) or white (1). This of course counts for all subsequent layers too.
  
  > What is the correct way to model an artificial neuron that outputs 0 when 1 is input and outputs 1 when 0 is input?
  
  As @Minh-Long Luu points out, you probably don't want or need a neural network to do this...
  In fact, I'm not even sure if this is possible (assuming training via backprop) because it sounds like you're after a [heaviside step-function](https://en.wikipedia.org/wiki/Heaviside_step_function) for the non-linear activation, which would be non-continuous and therefore non-differentiable, making it impossible to train the NN with stochastic gradient descent.
  Perhaps a flipped [sigmoid](https://en.wikipedia.org/wiki/Sigmoid_function) centered at 0.5 would be more appropriate?
  
  > **Comments:**
  > **username_3**: I would expect a single sigmoid neuron trained with gradient descent to converge to that Heaviside step, assuming no regularization.  
  
  ---
  
  ### Answer by username_3 on 2023-02-16. Score: 0
  
  This inverter is an interesting initial application of an artificial neuron.
  
  Yes, you need a negative weight and... no, the sigmoid suffices if you do with a close enough approximation.
  
  See how an unrestricted optimization could progress from a very inexact approximation to the Heaviside step in the limit, flipped and shifted as Giulio suggests.
  
  [![enter image description here](https://i.sstatic.net/4gljW.gif)](https://i.sstatic.net/4gljW.gif)
  
  If you want to test other numbers, here is the [WolframAlpha link](https://www.wolframalpha.com/input?i=1%2F%281%2Bexp%28-100%281%2F2-x%29%29%29%3B%201%2F%281%2Bexp%28-10%281%2F2-x%29%29%29%3B%201%2F%281%2Bexp%28-%281%2F2-x%29%29%29).
  
  
  ---
  
  