Answers.Body: <p>I'm unable to comment as I don't have "50 reputation", but I was wondering if one could use the 7 tablebase as part of the answer for an 8 tablebase.  Did they do that for the 7 tablebase?</p>
  
  <p>That is, when determining the solution for 8 pieces, stop if you get to 7 pieces and just refer to the 7 tablebase.  Granted, even with that it would be huge because there would still be a lot of combinations <strong>before</strong> getting to 7 pieces, but less than it would if you wanted to figure it out without referencing the 7 tablebase?</p>
  
  <p>If the solution used a method of going in the other direction, then maybe it doesn't help?</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-31T22:43:40.580Z
Answers.Text: The tablebase stores whether a position is Win , Draw or Loss; and also a metric (e.g. DTC - meaning number of moves until the position moves into another category). It doesn't store "a solution" as you suggest, instead the engine that uses the tablebase will look at all possible moves in a position, rank the metrics for each resulting position, and select a move based on the best metric. If the best move was a capture then the engine will look up the base for the new position to work out its next move
Answers.UserDisplayName: M.M
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-01T21:48:16.493Z
Answers.LastActivityDate: 2016-08-01T21:48:16.493Z
Answers.UserDisplayName: Eric
Answers.Body: <p>I am a developer currently working on software to compute endgame tablebases. The real bottleneck with calculating positions is not memory per se but the working memory and computing cycles needed to calculate a network for an exponentially-growing set of positions.</p>
  <p>The number of positions grows exponentially according to the number of men on the board; this means that the working memory needed (whether it be RAM or hard drive reads, which are less expensive and more scalable but significantly slower) increases proportionately to <code>A^N</code> where <code>A</code> is a scaling constant (somewhat less than 64) and <code>N</code> is the number of men in the tablebase. The Lomonosov Supercomputer used <a href="https://chessprogramming.wikispaces.com/Lomonosov+Supercomputer" rel="nofollow noreferrer">92 TiB of RAM</a>, which is sufficient for a 7-man tablebase; as dfan said in his comment, the jump is too big.</p>
  <p>Furthermore, as more nodes are added to the tablebase the number of calculations increases faster than linearly to the number of nodes. For instance, accessing data in a <a href="https://en.wikipedia.org/wiki/Binary_search_tree" rel="nofollow noreferrer">Binary Search Tree</a> (BST) is proportional to <code>ln(M)</code> where <code>M</code> is the number of nodes in the tree; since <code>M = A^N</code>, to access each node of the tree once, <code>A^N * ln(A^N)</code> operations are needed. There are several steps in the <a href="https://chessprogramming.wikispaces.com/Retrograde+Analysis" rel="nofollow noreferrer">tablebase retrograde evaluation algorithm</a> that scale similarly.</p>
  <p>In short, memory increases exponentially and the number of operations increases <em>faster than exponentially</em> as more men are added. Computing has not advanced this far in (at the time of writing) 6 years.</p>
  <h2>Edit</h2>
  <p>Since this answer is getting a lot of traffic and I re-read my answer, I feel the need to add some clarification. Endgame tablebases are created with retrograde analysis, so deterministically calculating the outcome of one position requires knowledge of enough possible successor positions. As we know, this graph mushrooms quickly. While retrograde analysis is definitely parallelizable, one needs an enormous amount of working memory. However, it is common practice to allocate only 10s of GB of RAM to a single processor, so positions must be communicated over a network. This would require a lot of bandwidth, and multiple processors contending to make changes to the same table state would create synchronization problems that add overhead. Positions must be stored in RAM because disk read latency is horrendous (on the order of milliseconds instead of nanoseconds for cache or microseconds for RAM). It's possible some amount of pipelining or caching could amortize poor latency, but this is still a hard problem to solve at the required scale.</p>
  <p>Tl;dr even with a big supercomputer, modern computer architecture is not amenable to this type of problem.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-14T00:35:13.620Z
Answers.LastActivityDate: 2020-11-16T06:17:23.073Z
Answers.LastEditDate: 2020-11-16T06:17:23.073Z
Answers.UserDisplayName: bcdan
Body: <p>Due to <a href="http://chessok.com/?page_id=27966" rel="noreferrer">Wikipedia: Endgame tablebase</a> the Lomonosov endgame tablebase is still the biggest tablebase (with a size of 140TB). Now this tablebase was computed in the year 2012 with the Lomonosov 1 (<a href="https://www.top500.org/list/2012/11/" rel="noreferrer">Top500-list November 2012, rank 26</a>). The Lomonosov 1 had 78.660 cores with ~900 TFlop/s (peak: 1.700 TFlop/s). When I have a look on the most <a href="https://www.top500.org/list/2016/06/" rel="noreferrer">recent Top500-List (June 2016)</a> I see that the Lomonosov 2 is ranked 41 in the world with ~2.100 TFlop/s (peak: ~2.900 TFlop/s), the No1-ranked super-computer has 93.000 TFlop/s.</p>
  <p>So I wonder why the Lomonosov-tablebase after 4 years still is the biggest tablebase. Shouldn't it be easy with the growing computer-power to calculate an 8-men-tablebase? Or is the step from 7 to 8 just too big? Maybe the size would go up exponentially so that standard-storage-systems just cannot handle this? Would appreciate all information about this theme.</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-07-24T21:11:27.410Z
Comments.Text: The step from 7 to 8 is just too big. According to that Wikipedia article, it takes 7 GB to store the tablebases up to 5 men (in Nalimov format), 1.2 TB to store the tablebases up to 6 men, and 140 TB to store the tablebases up to 7 men. So storing an 8-man tablebase will require well over a petabyte of storage.
Comments.UserDisplayName: dfan
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-07-25T01:15:10.753Z
Comments.Text: @dfan Your comments should be an answer.
Comments.UserDisplayName: SmallChess
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-07-25T01:37:42.587Z
Comments.Text: @StudentT I didn't think it was definitive enough.
Comments.UserDisplayName: dfan
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-07-25T06:28:44.093Z
Comments.Text: They could do 7-men with opposing pawns on same file configs counted as "1" rather than "2", or something. This would only about 3x as large as current. Also, memory (across all nodes) is more important than speed ("TFlops", not precisely the best measure) I think.
Comments.UserDisplayName: Post-It-Note
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-31T22:52:38.027Z
Comments.Text: according to [this page](http://tb7.chessok.com/articles/Top8DTM_eng) it's estimated that computing 8-man TB will require 10PB of storage and 50TB of RAM.  That's not to mention the time required -- with that volume of storage and random access requirements, much technology is needed to manage the storage access times.
Comments.UserDisplayName: M.M
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-09-01T06:11:57.780Z
Comments.Text: @M.M On the other hand, in much the same way that computing Pi or large primes was used to test and 'break in' supercomputers during the 80s and 90s, it seems as though chess tablebases are an excellent toy instance of the problems that are at the frontier of what computing is today - using large amounts of memory to manipulate even larger Big Data sets.
Comments.UserDisplayName: Steven Stadnicki
Tags: <engines><tablebases>
Title: How is it possible that the Lomonosov-endgame-tablebase from 2012 still is the most powerful tablebase?
ThreadText: # How is it possible that the Lomonosov-endgame-tablebase from 2012 still is the most powerful tablebase?
  
  (Asked by: username_0 on 2016-07-24)
  
  Due to [Wikipedia: Endgame tablebase](http://chessok.com/?page_id=27966) the Lomonosov endgame tablebase is still the biggest tablebase (with a size of 140TB). Now this tablebase was computed in the year 2012 with the Lomonosov 1 ([Top500-list November 2012, rank 26](https://www.top500.org/list/2012/11/)). The Lomonosov 1 had 78.660 cores with ~900 TFlop/s (peak: 1.700 TFlop/s). When I have a look on the most [recent Top500-List (June 2016)](https://www.top500.org/list/2016/06/) I see that the Lomonosov 2 is ranked 41 in the world with ~2.100 TFlop/s (peak: ~2.900 TFlop/s), the No1-ranked super-computer has 93.000 TFlop/s.
  
  So I wonder why the Lomonosov-tablebase after 4 years still is the biggest tablebase. Shouldn't it be easy with the growing computer-power to calculate an 8-men-tablebase? Or is the step from 7 to 8 just too big? Maybe the size would go up exponentially so that standard-storage-systems just cannot handle this? Would appreciate all information about this theme.
  
  **Tags:** engines, tablebases
  
  > **Comments:**
  > **username_1**: The step from 7 to 8 is just too big. According to that Wikipedia article, it takes 7 GB to store the tablebases up to 5 men (in Nalimov format), 1.2 TB to store the tablebases up to 6 men, and 140 TB to store the tablebases up to 7 men. So storing an 8-man tablebase will require well over a petabyte of storage.  
  > **username_2**: @username_1 Your comments should be an answer.  
  > **username_1**: @StudentT I didn't think it was definitive enough.  
  > **username_3**: They could do 7-men with opposing pawns on same file configs counted as "1" rather than "2", or something. This would only about 3x as large as current. Also, memory (across all nodes) is more important than speed ("TFlops", not precisely the best measure) I think.  
  > **username_4**: according to [this page](http://tb7.chessok.com/articles/Top8DTM_eng) it's estimated that computing 8-man TB will require 10PB of storage and 50TB of RAM.  That's not to mention the time required -- with that volume of storage and random access requirements, much technology is needed to manage the storage access times.  
  > **username_5**: @username_4 On the other hand, in much the same way that computing Pi or large primes was used to test and 'break in' supercomputers during the 80s and 90s, it seems as though chess tablebases are an excellent toy instance of the problems that are at the frontier of what computing is today - using large amounts of memory to manipulate even larger Big Data sets.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_7 on 2018-07-14. Score: 9 | Accepted answer
  
  I am a developer currently working on software to compute endgame tablebases. The real bottleneck with calculating positions is not memory per se but the working memory and computing cycles needed to calculate a network for an exponentially-growing set of positions.
  
  The number of positions grows exponentially according to the number of men on the board; this means that the working memory needed (whether it be RAM or hard drive reads, which are less expensive and more scalable but significantly slower) increases proportionately to `A^N` where `A` is a scaling constant (somewhat less than 64) and `N` is the number of men in the tablebase. The Lomonosov Supercomputer used [92 TiB of RAM](https://chessprogramming.wikispaces.com/Lomonosov+Supercomputer), which is sufficient for a 7-man tablebase; as username_1 said in his comment, the jump is too big.
  
  Furthermore, as more nodes are added to the tablebase the number of calculations increases faster than linearly to the number of nodes. For instance, accessing data in a [Binary Search Tree](https://en.wikipedia.org/wiki/Binary_search_tree) (BST) is proportional to `ln(M)` where `M` is the number of nodes in the tree; since `M = A^N`, to access each node of the tree once, `A^N * ln(A^N)` operations are needed. There are several steps in the [tablebase retrograde evaluation algorithm](https://chessprogramming.wikispaces.com/Retrograde+Analysis) that scale similarly.
  
  In short, memory increases exponentially and the number of operations increases *faster than exponentially* as more men are added. Computing has not advanced this far in (at the time of writing) 6 years.
  
  ## Edit
  
  Since this answer is getting a lot of traffic and I re-read my answer, I feel the need to add some clarification. Endgame tablebases are created with retrograde analysis, so deterministically calculating the outcome of one position requires knowledge of enough possible successor positions. As we know, this graph mushrooms quickly. While retrograde analysis is definitely parallelizable, one needs an enormous amount of working memory. However, it is common practice to allocate only 10s of GB of RAM to a single processor, so positions must be communicated over a network. This would require a lot of bandwidth, and multiple processors contending to make changes to the same table state would create synchronization problems that add overhead. Positions must be stored in RAM because disk read latency is horrendous (on the order of milliseconds instead of nanoseconds for cache or microseconds for RAM). It's possible some amount of pipelining or caching could amortize poor latency, but this is still a hard problem to solve at the required scale.
  
  Tl;dr even with a big supercomputer, modern computer architecture is not amenable to this type of problem.
  
  
  ---
  
  ### Answer by username_6 on 2016-08-01. Score: 3
  
  I'm unable to comment as I don't have "50 reputation", but I was wondering if one could use the 7 tablebase as part of the answer for an 8 tablebase. Did they do that for the 7 tablebase?
  
  That is, when determining the solution for 8 pieces, stop if you get to 7 pieces and just refer to the 7 tablebase. Granted, even with that it would be huge because there would still be a lot of combinations **before** getting to 7 pieces, but less than it would if you wanted to figure it out without referencing the 7 tablebase?
  
  If the solution used a method of going in the other direction, then maybe it doesn't help?
  
  > **Comments:**
  > **username_4**: The tablebase stores whether a position is Win , Draw or Loss; and also a metric (e.g. DTC - meaning number of moves until the position moves into another category). It doesn't store "a solution" as you suggest, instead the engine that uses the tablebase will look at all possible moves in a position, rank the metrics for each resulting position, and select a move based on the best metric. If the best move was a capture then the engine will look up the base for the new position to work out its next move  
  
  ---
  
  