Answers.Body: <p>I will extend a bit on Scounged's comment, which in principle suffices as an answer.</p>
  <p>Indeed, each neuron is nothing more than a little funtion computer that takes all inputs, multiplies them with a weighting, and &quot;fires&quot; if the resultat is above some level (e.g. by a sigmoid, but 0/1 would work as well - anything nonlinear works).</p>
  <p>The ineffability of NN always has been a main criticism against them. It is typical &quot;emergent&quot; behavior, and any NN that is large enough can no longer be split into sensible sub-units, even if the emulated function can. (It is very possible that a NN that simulates the KR/K mating technique is still simple enough for such an analysis.)</p>
  <p>&quot;Understanding NN&quot; is cutting-edge research. &quot;Understanding&quot; as in &quot;understanding the math behind it&quot; is <a href="https://towardsdatascience.com/understanding-neural-networks-19020b758230?gi=c3bdb7d44a6f" rel="nofollow noreferrer">common knowledge</a> and accessible to anyone with average math and computer science abilities, but &quot;understanding&quot; as in &quot;understanding how it solves a task&quot; is <a href="https://www.kdnuggets.com/2020/07/understanding-neural-networks-think.html" rel="nofollow noreferrer">difficult</a>.</p>
  <p>It would be an interesting experiment to kill a specific neuron of Leela and see what happens, but I bet you could kill a specific neuron in <em>my</em> brain and nothing will happen to my ELO. Specifically, I do <em>not</em> expect that Leela suddenly neglects king safety, since Leela already has too many neurons.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-05T16:57:04.377Z
Answers.LastActivityDate: 2022-02-05T16:57:04.377Z
Answers.UserDisplayName: Hauke Reddmann
Body: <p>I am referring to <a href="https://www.chessprogramming.org/File:StockfishNNUELayers.png" rel="nofollow noreferrer">this</a> picture.</p>
  <p><a href="https://i.sstatic.net/9ZseU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9ZseU.png" alt="enter image description here" /></a></p>
  <p>What do the individual neurons in the ANN do? Do they represent criteria for the evaluation, like for example material, pawn structure, king safety and so on?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-05T13:42:04.227Z
Comments.Text: I think the best way to answer this question is with an analogy: what do the individual neurons in a human brain do? In general it will be more or less impossible to understand a neural network by looking at each individual node trying to decipher what that node does exactly, since all an individual node does is emit an output signal based on the inputs coming into it.
Comments.UserDisplayName: Scounged
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-05T14:33:39.073Z
Comments.Text: They simply calculate an activation function (sigmoid, relu, etc).
Comments.UserDisplayName: SecretAgentMan
Tags: <stockfish>
Title: Neurons Stockfish NNUE
ThreadText: # Neurons Stockfish NNUE
  
  (Asked by: username_0 on 2022-02-05)
  
  I am referring to [this](https://www.chessprogramming.org/File:StockfishNNUELayers.png) picture.
  
  [![enter image description here](https://i.sstatic.net/9ZseU.png)](https://i.sstatic.net/9ZseU.png)
  
  What do the individual neurons in the ANN do? Do they represent criteria for the evaluation, like for example material, pawn structure, king safety and so on?
  
  **Tags:** stockfish
  
  > **Comments:**
  > **username_1**: I think the best way to answer this question is with an analogy: what do the individual neurons in a human brain do? In general it will be more or less impossible to understand a neural network by looking at each individual node trying to decipher what that node does exactly, since all an individual node does is emit an output signal based on the inputs coming into it.  
  > **username_2**: They simply calculate an activation function (sigmoid, relu, etc).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2022-02-05. Score: 4 | Accepted answer
  
  I will extend a bit on username_1's comment, which in principle suffices as an answer.
  
  Indeed, each neuron is nothing more than a little funtion computer that takes all inputs, multiplies them with a weighting, and "fires" if the resultat is above some level (e.g. by a sigmoid, but 0/1 would work as well - anything nonlinear works).
  
  The ineffability of NN always has been a main criticism against them. It is typical "emergent" behavior, and any NN that is large enough can no longer be split into sensible sub-units, even if the emulated function can. (It is very possible that a NN that simulates the KR/K mating technique is still simple enough for such an analysis.)
  
  "Understanding NN" is cutting-edge research. "Understanding" as in "understanding the math behind it" is [common knowledge](https://towardsdatascience.com/understanding-neural-networks-19020b758230?gi=c3bdb7d44a6f) and accessible to anyone with average math and computer science abilities, but "understanding" as in "understanding how it solves a task" is [difficult](https://www.kdnuggets.com/2020/07/understanding-neural-networks-think.html).
  
  It would be an interesting experiment to kill a specific neuron of Leela and see what happens, but I bet you could kill a specific neuron in *my* brain and nothing will happen to my ELO. Specifically, I do *not* expect that Leela suddenly neglects king safety, since Leela already has too many neurons.
  
  
  ---
  
  