Answers.Body: <p>Here is a working solution to the problem.</p>
  <p>The first step is to compute evaluation strength as normal for mates &amp; centipawns.</p>
  <ol>
  <li>Take <code>sigmoid(centipawn / downscale_factor)</code> for every centipawn value. The downscale factor allows you to vary the spread. This puts all centipawn values into a <strong>[1, 0]</strong> normalised range.</li>
  <li>Take <code>sigmoid(-1 * (mate_value / downscale_factor)</code> for every white mate. The downscale factor allows you to vary the spread. Mate value is the mate number like mate in 3. This give you the white mates in <strong>[0.5, 0]</strong> range.</li>
  <li>Repeat step 2 for black mates.</li>
  <li>Now create a list of normalised values that range from <strong>[1.5, -0.5]</strong> for all of your values. We get this by simply adding 1 to all the white mate values from before, and taking away from 0 the values from the black mate.</li>
  </ol>
  <p>At this stage, we now have a normalised <strong>[1.5, -0.5]</strong> range of values including mates &amp; centipawns. Note, that the range includes exponential effects in multiple places (i.e. there is a large exponential decay as we go from 1.5 to 1. There is a decay from -0.5 to 0. And a decay outwards from 0 (neutral). This is exactly what we want.</p>
  <ol start="5">
  <li>Convert this <strong>[1.5, -0.5]</strong> range into a [1, 0] range using linear scalar.</li>
  </ol>
  <p>Calculating difficulty / variance.
  This section is a little domain specific to Chess game I made where you sort 4 chess moves.</p>
  <ol>
  <li>Take the strengths from before and your list of move evaluations</li>
  <li>Calculate the adjacent diffs between each value</li>
  <li>Remove all 0 value adjacent diff values (because if its 0 it means two adjacent moves have same evaluation and that means its easy to sort because they're correct in either position).</li>
  <li>Take <code>harmonic_mean</code> on value so there is strong bias towards smaller gaps.</li>
  </ol>
  <p>Done!</p>
  <p>For anyone interested in a code example of this, you can see:
  <a href="https://github.com/mobeigi/chessort/tree/main/server/chessortserver/utils/engine" rel="nofollow noreferrer">https://github.com/mobeigi/chessort/tree/main/server/chessortserver/utils/engine</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-05T19:58:26.703Z
Answers.LastActivityDate: 2024-08-05T19:58:26.703Z
Answers.UserDisplayName: Mo Beigi
Answers.Body: <p>Instead of var(x_i), maybe try var(y_i) where y_i =sign(x_i)*log(abs(1+x_i)). Using logarithmic scale means the answer is more sensitive to the smaller values than the larger ones.</p>
  <p>Using sign &amp; absolute value ensures that positive and negative numbers are handled equally. Adding 1 before taking the log means 0 maps to 0.</p>
  <p>This doesn’t directly answer the question of how to map checkmates, but it does mean that the way you do so matters less!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-31T17:08:35.457Z
Answers.Text: I headed in a different direction by using `exponential decays` for mate values and passing it into a `sigmoid` function. Then finally taking a `harmonic mean`. Its not perfect by any means but its something (link to code in question comment if interested). I hope I can improve it shortly to be more accurate / realistic.
Answers.UserDisplayName: Mo Beigi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-31T04:20:01.930Z
Answers.LastActivityDate: 2024-07-31T04:20:01.930Z
Answers.UserDisplayName: Laska
Answers.Body: <p>Before even calculating the second order moment of this, are you sure you are having the same ordered domain to do the statistics on?</p>
  <p>What is the ordering relation between #n and #m?  I guess one could impose some exaggeration of the 50 move rules that shorter is always better (without any other information, about players and board).</p>
  <p>But then what kind of measure to use in a mixture of such outcome quantities, one of categorical ordering, and the other from forecasting learned or programmed leaf evaluation function of outcome odds from best play continuation (if having such conversion from elsewhere on some statistical basis, if the evaluation non-terminal value is not itself derived from a probabilistic mathematical model, say LC0, or A0, using RL training).</p>
  <p>When integrating over those leaf evaluation, there might be a need for some floating parameter, and you might have to do, like fishtest, do a global optimization search on that moment definition. Average or variance, they all need such definition.  Between the terminal outcome scale. And the forecasting one.  By forecasting, I mean the non-terminal leaf evaluations.  e.g. 1,3,3,4,9 might be a possible forecasting system. But we found that it would not always be enough. Etc.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-01T13:13:18.617Z
Answers.Text: I don't have a good answer to what the ordering relation between `#n` and `#m`. We are making a broad and general assumption that lower ones #1 is easier than #30. I'm also introducing a strong exponential decay that drop off rapidly after #7 (on anecdotal evidence that strong human GM players can find a #7 but may struggle finding higher mates).
  
  Its not ideal for any really valuable work but for my use case, any rough approximation that isn't totally random / nonsense will do.
Answers.UserDisplayName: Mo Beigi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-01T00:47:09.533Z
Answers.LastActivityDate: 2024-08-01T00:47:09.533Z
Answers.UserDisplayName: dbdb
Body: <h2>Introduction</h2>
  <p>You are given a set of Chess move evaluations. They contain centipawn values and also mate values.</p>
  <p>For example, an example set: <code>[#1, +234, -17, #-4]</code></p>
  <p>The above example contains the evaluations for 4 random moves. The board position and moves themselves are not important.</p>
  <p>The problem I am trying to solve is to generate a variance for this set. I want to know how &quot;close&quot; together the evaluations are. I'd like to then normalize this final value into a reasonable range such as <code>0-100</code> where <code>0</code> is not close at all and <code>100</code> is extremely close (all moves have same strength).</p>
  <p>Using the statistics <a href="https://en.wikipedia.org/wiki/Variance" rel="nofollow noreferrer">Variance</a> formula, I am able to do this easily and well for just centipawn values. It works flawlessly.</p>
  <p>Some examples for centipawn values to illustrate the idea:</p>
  <div class="s-table-container"><table class="s-table">
  <thead>
  <tr>
  <th>Evaluation Set</th>
  <th>Variance</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>{300, 300, 300, 300}</td>
  <td>0.0</td>
  </tr>
  <tr>
  <td>{0, 0, 0, 0}</td>
  <td>0.0</td>
  </tr>
  <tr>
  <td>{300, 299, 298, 50}</td>
  <td>11625.688</td>
  </tr>
  <tr>
  <td>{300, 200, 100, 0}</td>
  <td>12500.0</td>
  </tr>
  <tr>
  <td>{500, 400, 300, 200}</td>
  <td>12500.0</td>
  </tr>
  <tr>
  <td>{600, 500, 400, 300}</td>
  <td>12500.0</td>
  </tr>
  <tr>
  <td>{300, 299, 100, 50}</td>
  <td>12912.688</td>
  </tr>
  <tr>
  <td>{300, 299, 298, 0}</td>
  <td>16763.188</td>
  </tr>
  <tr>
  <td>{500, 400, 300, 0}</td>
  <td>35000.0</td>
  </tr>
  <tr>
  <td>{600, 500, 400, 0}</td>
  <td>51875.0</td>
  </tr>
  <tr>
  <td>{1000, 50, -200, -750}</td>
  <td>400625.0</td>
  </tr>
  </tbody>
  </table></div>
  <p>In the above examples, you can see sets of 4 moves in order from very close to not close. The first set <code>{300, 300, 300, 300}</code> has variance of 0 because all items are the same. This should map to <code>100</code> for my normalized output.</p>
  <p><code>{500, 400, 300, 200}</code> and <code>{600, 500, 400, 300}</code> are equivalent because they all spread out the same (even though one set has stronger moves in each position).</p>
  <p><code>{1000, 50, -200, -750}</code> has a very large variance and thus is very spread out. This should map to near 0. Obviously variance can be unbound but I have an upper bound I enforce (currently using <strong>400<sup>2</sup></strong>).</p>
  <h2>Problem</h2>
  <p>How can I introduce check mates into my variance equation? I am struggling with this.</p>
  <p>Obviously a <code>#1</code> by itself should be very 'not close' in an example like <code>[#1, +300, +100, -100]</code>.</p>
  <p>However, what if there are two mates in the set like <code>[#1, #2, +300, +100]</code>? A <code>#1</code> and <code>#2</code> are (very?) close to each other but there are also centipawn values to compare.</p>
  <p>What about this scenario: <code>[#1, #2, +300, #-1]</code> which contains all the different possibilities mates for white/black + centipawns.</p>
  <p>Hopefully someone has ideas or can lead me in the right direction.</p>
  <h2>Context</h2>
  <p>For additional context, I am making a fun Chess game and need a way to compare the difficulty of sets of 4 moves. Basically answer the question &quot;How strong are these 4 moves?&quot;.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-30T15:34:03.890Z
Comments.Text: You should simply assign a large arbitrary value to checkmates. Anyway I see a major problem with the concept of variance applied to move evaluations. A set like {+9, +15} may seem to have more variance than the set {-1, +1} but in practical terms it doesn't. Specially if you want to relate it to the difficulty of a position
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-30T15:55:47.093Z
Comments.Text: I tried assigning a large value to checkmates (like `32000-N`) but it dominates the variance equation which isn't correct. Thanks for clarification on my assumption on variance. You are right, evals closer to 0 are  "more varied" in the context of difficulty of position.
Comments.UserDisplayName: Mo Beigi
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-31T17:06:51.093Z
Comments.Text: I spend a lot of time on this and this is the best I could come up with:
  https://gist.github.com/mobeigi/2abe90d5f6d6cb2eb3ecfb67b29fb326
  
  Its not perfect by any means but it somewhat works (and my brain hurts thinking about this so much!).
Comments.UserDisplayName: Mo Beigi
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-31T18:14:20.993Z
Comments.Text: Please keep in mind that comments are a sign of bad code, not good one. Let the code speak for itself!
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-01T13:11:00.650Z
Comments.Text: I tend to over comment when I don't understand the domain :D
Comments.UserDisplayName: Mo Beigi
Tags: <analysis><programming><evaluation><mathematics>
Title: How to compute variance of a set of Chess move evaluations
ThreadText: # How to compute variance of a set of Chess move evaluations
  
  (Asked by: username_0 on 2024-07-30)
  
  ## Introduction
  
  You are given a set of Chess move evaluations. They contain centipawn values and also mate values.
  
  For example, an example set: `[#1, +234, -17, #-4]`
  
  The above example contains the evaluations for 4 random moves. The board position and moves themselves are not important.
  
  The problem I am trying to solve is to generate a variance for this set. I want to know how "close" together the evaluations are. I'd like to then normalize this final value into a reasonable range such as `0-100` where `0` is not close at all and `100` is extremely close (all moves have same strength).
  
  Using the statistics [Variance](https://en.wikipedia.org/wiki/Variance) formula, I am able to do this easily and well for just centipawn values. It works flawlessly.
  
  Some examples for centipawn values to illustrate the idea:
  
  | Evaluation Set | Variance |
  | --- | --- |
  | {300, 300, 300, 300} | 0.0 |
  | {0, 0, 0, 0} | 0.0 |
  | {300, 299, 298, 50} | 11625.688 |
  | {300, 200, 100, 0} | 12500.0 |
  | {500, 400, 300, 200} | 12500.0 |
  | {600, 500, 400, 300} | 12500.0 |
  | {300, 299, 100, 50} | 12912.688 |
  | {300, 299, 298, 0} | 16763.188 |
  | {500, 400, 300, 0} | 35000.0 |
  | {600, 500, 400, 0} | 51875.0 |
  | {1000, 50, -200, -750} | 400625.0 |
  
  In the above examples, you can see sets of 4 moves in order from very close to not close. The first set `{300, 300, 300, 300}` has variance of 0 because all items are the same. This should map to `100` for my normalized output.
  
  `{500, 400, 300, 200}` and `{600, 500, 400, 300}` are equivalent because they all spread out the same (even though one set has stronger moves in each position).
  
  `{1000, 50, -200, -750}` has a very large variance and thus is very spread out. This should map to near 0. Obviously variance can be unbound but I have an upper bound I enforce (currently using **4002**).
  
  ## Problem
  
  How can I introduce check mates into my variance equation? I am struggling with this.
  
  Obviously a `#1` by itself should be very 'not close' in an example like `[#1, +300, +100, -100]`.
  
  However, what if there are two mates in the set like `[#1, #2, +300, +100]`? A `#1` and `#2` are (very?) close to each other but there are also centipawn values to compare.
  
  What about this scenario: `[#1, #2, +300, #-1]` which contains all the different possibilities mates for white/black + centipawns.
  
  Hopefully someone has ideas or can lead me in the right direction.
  
  ## Context
  
  For additional context, I am making a fun Chess game and need a way to compare the difficulty of sets of 4 moves. Basically answer the question "How strong are these 4 moves?".
  
  **Tags:** analysis, programming, evaluation, mathematics
  
  > **Comments:**
  > **username_1**: You should simply assign a large arbitrary value to checkmates. Anyway I see a major problem with the concept of variance applied to move evaluations. A set like {+9, +15} may seem to have more variance than the set {-1, +1} but in practical terms it doesn't. Specially if you want to relate it to the difficulty of a position  
  > **username_0**: I tried assigning a large value to checkmates (like `32000-N`) but it dominates the variance equation which isn't correct. Thanks for clarification on my assumption on variance. You are right, evals closer to 0 are  "more varied" in the context of difficulty of position.  
  > **username_0**: I spend a lot of time on this and this is the best I could come up with: https://gist.github.com/username_0/2abe90d5f6d6cb2eb3ecfb67b29fb326  Its not perfect by any means but it somewhat works (and my brain hurts thinking about this so much!).  
  > **username_1**: Please keep in mind that comments are a sign of bad code, not good one. Let the code speak for itself!  
  > **username_0**: I tend to over comment when I don't understand the domain :D  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_0 on 2024-08-05. Score: 1
  
  Here is a working solution to the problem.
  
  The first step is to compute evaluation strength as normal for mates & centipawns.
  
  1. Take `sigmoid(centipawn / downscale_factor)` for every centipawn value. The downscale factor allows you to vary the spread. This puts all centipawn values into a **[1, 0]** normalised range.
  2. Take `sigmoid(-1 * (mate_value / downscale_factor)` for every white mate. The downscale factor allows you to vary the spread. Mate value is the mate number like mate in 3. This give you the white mates in **[0.5, 0]** range.
  3. Repeat step 2 for black mates.
  4. Now create a list of normalised values that range from **[1.5, -0.5]** for all of your values. We get this by simply adding 1 to all the white mate values from before, and taking away from 0 the values from the black mate.
  
  At this stage, we now have a normalised **[1.5, -0.5]** range of values including mates & centipawns. Note, that the range includes exponential effects in multiple places (i.e. there is a large exponential decay as we go from 1.5 to 1. There is a decay from -0.5 to 0. And a decay outwards from 0 (neutral). This is exactly what we want.
  
  5. Convert this **[1.5, -0.5]** range into a [1, 0] range using linear scalar.
  
  Calculating difficulty / variance.
  This section is a little domain specific to Chess game I made where you sort 4 chess moves.
  
  1. Take the strengths from before and your list of move evaluations
  2. Calculate the adjacent diffs between each value
  3. Remove all 0 value adjacent diff values (because if its 0 it means two adjacent moves have same evaluation and that means its easy to sort because they're correct in either position).
  4. Take `harmonic_mean` on value so there is strong bias towards smaller gaps.
  
  Done!
  
  For anyone interested in a code example of this, you can see:
  <https://github.com/username_0/chessort/tree/main/server/chessortserver/utils/engine>
  
  
  ---
  
  ### Answer by username_2 on 2024-07-31. Score: 0
  
  Instead of var(x\_i), maybe try var(y\_i) where y\_i =sign(x\_i)\*log(abs(1+x\_i)). Using logarithmic scale means the answer is more sensitive to the smaller values than the larger ones.
  
  Using sign & absolute value ensures that positive and negative numbers are handled equally. Adding 1 before taking the log means 0 maps to 0.
  
  This doesn’t directly answer the question of how to map checkmates, but it does mean that the way you do so matters less!
  
  > **Comments:**
  > **username_0**: I headed in a different direction by using `exponential decays` for mate values and passing it into a `sigmoid` function. Then finally taking a `harmonic mean`. Its not perfect by any means but its something (link to code in question comment if interested). I hope I can improve it shortly to be more accurate / realistic.  
  
  ---
  
  ### Answer by username_3 on 2024-08-01. Score: 0
  
  Before even calculating the second order moment of this, are you sure you are having the same ordered domain to do the statistics on?
  
  What is the ordering relation between #n and #m? I guess one could impose some exaggeration of the 50 move rules that shorter is always better (without any other information, about players and board).
  
  But then what kind of measure to use in a mixture of such outcome quantities, one of categorical ordering, and the other from forecasting learned or programmed leaf evaluation function of outcome odds from best play continuation (if having such conversion from elsewhere on some statistical basis, if the evaluation non-terminal value is not itself derived from a probabilistic mathematical model, say LC0, or A0, using RL training).
  
  When integrating over those leaf evaluation, there might be a need for some floating parameter, and you might have to do, like fishtest, do a global optimization search on that moment definition. Average or variance, they all need such definition. Between the terminal outcome scale. And the forecasting one. By forecasting, I mean the non-terminal leaf evaluations. e.g. 1,3,3,4,9 might be a possible forecasting system. But we found that it would not always be enough. Etc.
  
  > **Comments:**
  > **username_0**: I don't have a good answer to what the ordering relation between `#n` and `#m`. We are making a broad and general assumption that lower ones #1 is easier than #30. I'm also introducing a strong exponential decay that drop off rapidly after #7 (on anecdotal evidence that strong human GM players can find a #7 but may struggle finding higher mates).  Its not ideal for any really valuable work but for my use case, any rough approximation that isn't totally random / nonsense will do.  
  
  ---
  
  