Answers.Body: <p>I think it's best if I elaborate on your second point with an example move in the game 1 between AlphaZero and Stockfish which also served to satisfy my curiosity today.</p>
  
  <blockquote>
    <p>the time limit of 1 min/move (How would this disadvantage Stockfish?)</p>
  </blockquote>
  
  <p>Stockfish's performance is dependent upon both the time limit <em>and</em> the hardware configuration, so just think of when someone doubles the CPU threads, then Stockfish needs less time (not necessarily half) to find the solution than it would with the first configuration.</p>
  
  <p><a href="https://www.chess.com/news/view/google-s-alphazero-destroys-stockfish-in-100-game-match" rel="noreferrer">On the first report that was posted on Chess.com</a> someone claimed that Stockfish was not playing optimally because he couldn't reproduce the same results using the same Stockfish on his computer. He said that on the position below (game 1 - move 11) Stockfish played Kg1-h1 (moved its king) which made no sense at all. On the other hand, stockfish on his computer showed a more developing move like Be3 (move the dark square bishop), lets look at the position:</p>
  
  <p><a href="https://raw.githubusercontent.com/abdolrezat/Stockfish/Figures/figures/move11.png" rel="noreferrer"><img src="https://raw.githubusercontent.com/abdolrezat/Stockfish/Figures/figures/move11.png" alt="Game 1 on move 11"></a></p>
  
  <p>Yes, it was a passive move and it seems that Stockfish should have played a more developing move. But he was wrong. Why? Because he ran Stockfish for 15 seconds, and if he had run it for an hour he would have gotten Kg1-h1 as the best move in that position. Stockfish changes it's decision when it analyses all the possible moves in more depth. Here's <a href="https://www.chess.com/news/view/google-s-alphazero-destroys-stockfish-in-100-game-match" rel="noreferrer">what I originally said in my reply</a>:</p>
  
  <p>I ran the latest stockfish on the position (at move 11):</p>
  
  <ul>
  <li>At first, It gives b4 as the optimal move when the engine is running for about a minute. After that, it decides Be3 is better.</li>
  <li><p>But after 5 minutes on my hardware that runs on 1,400k nodes/s it will decide to go with Kh1 as the optimal move.</p></li>
  <li><p>In the paper, it is said that stockfish calculates 70,000k positions per second and is run for 1 minute per move, that's about 50 times my hardware, so I'll let mine run for 50 minutes... Kg1-h1 is still the choice for Stockfish.</p></li>
  </ul>
  
  <h2>Time limit is the key</h2>
  
  <p>In the above case, it probably didn't matter much if Stockfish ran for twice the time because the decision would have been the same, but on the next move it <strong>definitely would</strong>:</p>
  
  <p><a href="https://raw.githubusercontent.com/abdolrezat/Stockfish/Figures/figures/move12.png" rel="noreferrer"><img src="https://raw.githubusercontent.com/abdolrezat/Stockfish/Figures/figures/move12.png" alt="enter image description here"></a></p>
  
  <p>In this position, Stockfish chose to move the pawn on the left side (<strong>a4-a5</strong>). Let's say I have a computer that runs the Stockfish engine at a speed of 1,400k nodes per second, that's about 50 times lower than the Stockfish in the real game (<a href="https://arxiv.org/pdf/1712.01815.pdf" rel="noreferrer">In the paper</a>, it says 70,000k n/s). So I can simulate the game if I run it for 50 minutes at each move. Okay.</p>
  
  <p>I ran Stockfish analysis on the above position and I got the following results:</p>
  
  <ul>
  <li>Stockfish started out suggesting some moves, but after 6 minutes on my computer (corresponds to <strong>7.2 seconds</strong> on the Stockfish in the real game) <strong>it preferred a4-a5 just as the game went</strong>.</li>
  </ul>
  
  <p>That's good, but I kept it running for a complete 50 minutes in order to reach the computations of the Stockfish in the game that was allowed 1 minute:</p>
  
  <ul>
  <li>After 18 minutes the line changed back to <strong>Qc3</strong>.</li>
  <li>Again, after 24 minutes the best move was <strong>Ne3</strong>.</li>
  <li>After 27 minutes, it suggested <strong>Ng1</strong> all the way up to 56 minutes. After that, the optimal move was <strong>Nf3xe5</strong> but it's past the 50 minutes mark I mentioned before (<a href="https://raw.githubusercontent.com/abdolrezat/Stockfish/Figures/figures/120engineanalysis_LI.jpg" rel="noreferrer">you can see the engine analysis for an hour and 22 minutes in this screenshot</a>). </li>
  </ul>
  
  <p>The sad truth is that I believe Stockfish lost all its games because of the time limit. Stockfish gets a more in-depth search and evaluation as the time passes and in the game it wasn't allowed to use an opening book which makes it consider many moves in shallow depths. Note that in the actual game <em>a4-a5</em> was played which shows that (assuming it could evaluate 70 million positions per second) the Stockfish in the game didn't spend more than 21.6 seconds on the move. Otherwise, it would have changed its decision to those three other moves in the actual game. The reason for this is still unclear to me since my Stockfish was also consuming less memory (about ~130MB of RAM compared to the 1GB mentioned <a href="https://arxiv.org/pdf/1712.01815.pdf" rel="noreferrer">in the original paper</a>, assuming all of it goes to hash tables). </p>
  
  <hr>
  
  <h2>Conclusion</h2>
  
  <p>The hardware that ran Stockfish, as I pointed out, was at the very best 18 times faster than mine (Update: on a single core) based on the move I analyzed. I'm not sure if AlphaZero could really make use of such hardware to train its networks in 4 hours, I can only assume it's too low for a game like chess. Besides, AlphaZero spent those hours on learning which also includes building solid openings (and as the paper points out, preferences over certain openings). On the other hand, Stockfish was handicapped on openings, and it did not evaluate 70 million positions per second for 60 seconds on each move. </p>
  
  <p>As a final note, all the things I said were based on my assumptions. Of course, the outcome of AlphaZero and the games were super interesting to me. However, I would have loved to see a game where the Stockfish play was just like what I get on my computer, too. That is, more time and an opening book allowed. It's also easy to get the outputs of Stockfish analysis on every move, and I wish they release it in order to show how well it performed.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-09T22:40:38.373Z
Answers.Text: Regarding time limit, Figure 2 in AlphaZero paper shows opposite: Stockfish is better in lower budget, but scales worse, when more power is available.
  https://arxiv.org/pdf/1712.01815.pdf
Answers.UserDisplayName: old-ufo
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-10T12:20:09.973Z
Answers.Text: @old-ufo Thanks for pointing that out. As I said, the performance of Stockfish (and AlphaZero) is dependent on the hardware as well as the time limit. For Instance, if we gave Stockfish more hardware (and AlphaZero less) and regenerated that figure, its line could be transformed such that it always stays above the AlphaZero line. So I guess that's a good point in comparing the hardware for both systems which also answers the main question.
Answers.UserDisplayName: ReZzT
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-09T17:17:44.023Z
Answers.LastActivityDate: 2017-12-22T09:38:00.493Z
Answers.LastEditDate: 2017-12-22T09:38:00.493Z
Answers.UserDisplayName: ReZzT
Answers.Body: <p>One of the original authors of Stockfish answers the specific complaints you mentioned here:</p>
  
  <blockquote>
    <p>Meanwhile Chess.com also received a lengthy comment from one of the
    original Stockfish authors, Tord Romstad, which we'll give in full:</p>
    
    <blockquote>
      <p>The match results by themselves are not particularly meaningful
      because of the rather strange choice of time controls and Stockfish
      parameter settings: The games were played at a fixed time of 1
      minute/move, which means that Stockfish has no use of its time
      management heuristics (lot of effort has been put into making
      Stockfish identify critical points in the game and decide when to
      spend some extra time on a move; at a fixed time per move, the
      strength will suffer significantly). The version of Stockfish used is
      one year old, was playing with far more search threads than has ever
      received any significant amount of testing, and had way too small hash
      tables for the number of threads. I believe the percentage of draws
      would have been much higher in a match with more normal conditions.</p>
      
      <p>On the other hand, there is no doubt that AlphaZero could have played
      better if more work had been put into the project (although the "4
      hours of learning" mentioned in the paper is highly misleading when
      you take into account the massive hardware resources used during those
      4 hours). But in any case, Stockfish vs AlphaZero is very much a
      comparison of apples to orangutans. One is a conventional chess
      program running on ordinary computers, the other uses fundamentally
      different techniques and is running on custom designed hardware that
      is not available for purchase (and would be way out of the budget of
      ordinary users if it were).</p>
      
      <p>From another perspective, the apples vs orangutans angle is the most
      exciting thing about this: We now have two extremely different (both
      on the hardware and the software side) man-made entities that both
      display super-human chess playing abilities. That's much more
      interesting than yet another chess program that does the same thing as
      existing chess programs, just a little better. Furthermore, the
      adaptability of the AlphaZero approach to new domains opens exciting
      possibilities for the future.</p>
      
      <p>For chess players using computer chess programs as a tool, this
      breakthrough is unlikely to have a great impact, at least in the short
      term, because of the lack of suitable hardware for affordable prices.</p>
      
      <p>For chess engine programmers -- and for programmers in many other
      interesting domains -- the emergence of machine learning techniques
      that require massive hardware resources in order to be effective is a
      little disheartening. In a few years, it is quite possible that an
      AlphaZero like chess program can be made to run on ordinary computers,
      but the hardware resources required to <em>create</em> them will still be way
      beyond the budget of hobbyists or average sized companies. It is
      possible that an open source project with a large distributed network
      of computers run by volunteers could work, but the days of hundreds of
      unique chess engines, each with their own individual quirks and
      personalities, will be gone.</p>
    </blockquote>
  </blockquote>
  
  <p>Source: <a href="https://www.chess.com/news/view/alphazero-reactions-from-top-gms-stockfish-author" rel="noreferrer">https://www.chess.com/news/view/alphazero-reactions-from-top-gms-stockfish-author</a></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-10T02:00:29.240Z
Answers.LastActivityDate: 2017-12-10T02:00:29.240Z
Answers.UserDisplayName: Xonatron
Answers.Body: <p>Running on comparable hardware would be required if Google's end goal was to build a superior chess engine, but this exercise wasn't really about chess. Chess is just a convenient way to demonstrate the AI's ability to learn complicated tasks from scratch. If it can perform well against some vaguely reasonable configuration of Stockfish, it's checked the box. </p>
  
  <p>I predict the Google team will not spend much more effort on chess; instead, they will move on to other problems that AI has never been able to accomplish.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-19T00:40:58.693Z
Answers.Text: I've given +1 because I have the same feeling.
Answers.UserDisplayName: SmallChess
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-19T21:13:30.863Z
Answers.Text: Sounds probable, though I doubt they would have published it, if Alphazero had lost by only a small margin (meaning that it would still be of comparable strength to stockfish).
Answers.UserDisplayName: user1583209
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-20T00:32:58.933Z
Answers.Text: @user1583209 They probably ran it lots of times to figure out the minimum amount of learning time it needed to crush Stockfish. Then they did a final run and published those results.
Answers.UserDisplayName: TESch
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-18T21:16:53.113Z
Answers.LastActivityDate: 2017-12-18T21:16:53.113Z
Answers.UserDisplayName: TESch
Answers.Body: <p>Visit Talkchess Forum to know more, there is where you will find some 3000 programmers. 
  This was all a scam. 
  Alpha played on 30 times bigger hardware than SF, 4TPUs vs 64 cores. 
  4TPUs is around 1000 cores or even more. 
  Alpha had simulated opening book, trained on countless top GM winning games. 
  SF had very little hash. 
  TC was fixed at 1 minute per move, which is again detrimental to SF, which has advanced time management. 
  TPUs lack the SMP inefficiencies with more cores, so the hardware advantage was even bigger. 
  Etc, etc., so basically, this was just a huge publicity stunt on the part of Google. 
  Currently, Alpha is around 2800 on single core, so 400 elos below SF, and will not advanced much in the future, as, from now on, it will need advanced evaluation it will not be able to discover. 
  Concerning the 4-hours issue, well, LOL, this was 48 hours ago, so now Alpha is at 5000 elo? 
  Come on.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-09T15:28:34.487Z
Answers.Text: You seem to believe that AlphaZero does the same thing as Stockfish, only 1000 times faster because it used 1000 times stronger hardware. This is not true at all. It uses a very different approach and that approach is very resource intensive. In fact, during the match AlphaZero was evaluating 80 thousand positions per second while Stockfish was clocking at 70 million positions per second. Now tell me that AlphaZero won because of a stronger hardware. Of course on 64 CPU it would be slower and who knows how it would play but the point is that AlphaZero does it better, albeit at higher cost.
Answers.UserDisplayName: IA Petr Harasimovic
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-10T13:29:42.870Z
Answers.Text: SF hardware costs less than $10k, Alpha one more than $250k. Draw the conclusions yourself. Nps are meaningless, and every chess programmer knows that. You can do all kinds of tricks so that nps get lower, but that does not mean you will play stronger. I would like to see it implement that approach on SF hardware and SF its on Alpha hardware. Guess the result? +85 -0 =15 for SF. If they are so great, let them publish their code.
Answers.UserDisplayName: Lyudmil Tsvetkov
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-10T18:39:50.477Z
Answers.Text: "Alpha had simulated opening book, trained on countless top GM winning games." Simulated opening book, yes, but it trained on GM games? Do you have a source for this? My understanding was that Alpha was entirely bootstrapped.
Answers.UserDisplayName: Akavall
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-09T13:21:27.040Z
Answers.LastActivityDate: 2017-12-09T13:21:27.040Z
Answers.UserDisplayName: Lyudmil Tsvetkov
Answers.Body: <p>Stockfish is constrained to CPUs so it will never be able to scale to the level that GPUs are able.</p>
  
  <p>Gor matrix calculations GPUs scale with the n, while CPUs scale with  n<sup>3</sup>, these tensor cores are further optimized so it's likely even better performance as you scale.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-14T02:08:45.727Z
Answers.LastActivityDate: 2018-09-14T12:25:40.657Z
Answers.LastEditDate: 2018-09-14T12:25:40.657Z
Answers.UserDisplayName: Famous Jameis
Answers.Body: <p>First paragraph more detail, second short and simple answer third paragraph my opinions on the situation</p>
  
  <p>With AlphaZero the hardware has 0 effect on the strength of its play. It may take longer but not because it’s thinking. It’s a neural network, which means you feed it info in a vector (a single column table) it does simple math thru a giant tensor (a 3 or more dimensional table) then it spits out the answer. 
  Stockfish needs time to be good because it checks possible positions to see if a move is beneficial, so the longer it looks at the problem the more position/moves it can check. </p>
  
  <p>There isn’t really a comparable hardware setup. Because they have different needs, Stockfish needs to analyze more positions while AlphaZero just needs to make a move. And people are upset because AlphaZero's computer is technically much more powerful and they think they should be equal in that regard. But, AlphaZero doesn’t need that supercomputer after training.</p>
  
  <p>In my opinion it doesn’t matter what they give to either side, unless Stockfish has an unreasonable amount of time it will probably tie a few more games but in general a similar effect will happen. This is why I think this, Stockfish initially evaluates with pieces and their values while, Alpha played (probably) millions of games to realize what is important strategically. Which is why Alpha sacs a lot more than Stockfish would ever, but gains huge positional advantages.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-13T09:16:33.263Z
Answers.Text: That's just wrong. AlphaZero does tree search. More hardware makes it stronger. And it needs a ton of hardware to play better than stockfish.
Answers.UserDisplayName: BlindKungFuMaster
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-13T10:51:10.943Z
Answers.Text: Nothing in this post is correct...
Answers.UserDisplayName: SmallChess
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-14T14:33:47.240Z
Answers.Text: This system is based on 3 neuralnets and partial Monroe Carlo tree search, so you are correct about it using tree search. During play it uses 2 neural network principles developed in alphago of value and policy. Go is a game that can’t be done via computational power because it’s astronomically more complex than chess. So if I believe that it needs more computing power than a brute force algorithm then you are delusional. Or miss informed. The third neural net is used to try and excelerate the learning process by guessing the back propagation changes. Alphazeros power is in the nets not mcts.
Answers.UserDisplayName: Ezecal 
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-12T11:16:07.573Z
Answers.LastActivityDate: 2017-12-12T18:21:54.773Z
Answers.LastEditDate: 2017-12-12T18:21:54.773Z
Answers.UserDisplayName: Ezecal 
Answers.Body: <blockquote>
    <p>That's why I wonder whether there have been any attempts made to
    provide comparable hardware to both.</p>
  </blockquote>
  
  <p>This is Google you're talking about! So the answer is obviously "No".</p>
  
  <p>From <a href="https://arxiv.org/pdf/1712.01815.pdf" rel="noreferrer">the original paper</a> hardware used for initialising and training -</p>
  
  <blockquote>
    <p>Training proceeded for 700,000 steps (mini-batches of size 4,096)
    starting from randomly initialised parameters, using 5,000
    first-generation TPUs (15) to generate self-play games and 64
    second-generation TPUs to train the neural networks</p>
  </blockquote>
  
  <p>and hardware used for the games -</p>
  
  <blockquote>
    <p>AlphaZero and the previous AlphaGo Zero used a single machine with 4
    TPUs Stockfish and Elmo played at their strongest skill level using 64
    threads and a hash size of 1GB.</p>
  </blockquote>
  
  <p>So, AlphaZero used special hardware developed by Google. It used specialized Tensor Processor Units (TPUs) rather than general Central Processing Units (CPUs) as are available commercially.</p>
  
  <p>This is how Wikipedia describes the second generation TPUs they used -</p>
  
  <blockquote>
    <p>The second generation TPU was announced in May 2017. Google stated the
    first generation TPU design was memory bandwidth limited, and using 16
    GB of High Bandwidth Memory in the second generation design increased
    bandwidth to 600 GB/s and performance to 45 TFLOPS. The TPUs are then
    arranged into 4-chip 180 TFLOPS modules</p>
  </blockquote>
  
  <p>They used 4 TPUs for the games, so a processing power of 180 TFLOPS.
  Note TFLOPS = 1000 billion floating point operations per second.</p>
  
  <p>For comparison Intel's latest most powerful chip is the Core i9 Extreme Edition processor which clocks in at 1 TFLOP. A top of the line I7 that you would find in a gaming machine would typically be about 100 GFLOPs (i.e. one tenth of a TFLOP).</p>
  
  <p>I think it's fair to say that AlphaZero was using an 800 pound gorilla of a hardware configuration compared to Stockfishes mouse.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-08T18:47:40.827Z
Answers.Text: FLOPS stands for floating point operations per second. Floating point arithmetic is probably not used at all in the core algorithmes of Stockfish and AlphaZero. So the number of FLOPS is not really a meaningfull measure of processor speed relevant to the chess engine.
Answers.UserDisplayName: René Pijl
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-08T21:43:32.450Z
Answers.Text: On the contrary, I believe neural nets use floating point arithmetic quite intensively. (But of course your remark makes perfect sense and applies to Stockfish.)
Answers.UserDisplayName: Federico Poloni
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-08T21:53:59.323Z
Answers.Text: A far more apt comparison would be with a GPU; the NVIDIA Tesla V100 can do 120 TFLOPS, up from the previous generation (P100) which only did about 20. CPUs are optimized very differently from GPUs that do sheer volumes of numbers.
Answers.UserDisplayName: Nick T
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-08T13:12:40.250Z
Answers.LastActivityDate: 2017-12-08T13:12:40.250Z
Answers.UserDisplayName: Brian Towers
Body: <p>I understand that AlphaZero has to use a different kind of hardware than regular Stockfish. I would expect that the hardware has a large effect on engine strength. That's why I wonder whether there have been any attempts made to provide comparable hardware to both. Also what would "comparable" mean here?</p>
  
  <p>Specifically I read that people complain about:</p>
  
  <ul>
  <li>Stockfish being given only 1 GB of cache, and</li>
  <li>the time limit of 1 min/move (How would this disadvantage Stockfish?)</li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-09T17:15:15.703Z
Comments.Text: *Hash* size, not cache, apparently.
Comments.UserDisplayName: Federico Poloni
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-09T23:14:32.860Z
Comments.Text: I *strongly* suggest asking this on a technical forum (like the [AI Stack Exchange](https://ai.stackexchange.com/), perhaps titled "Fairness of evaluation in AlphaZero vs. Stockfish matches"), because the answers here are not good.
Comments.UserDisplayName: Veedrac
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-24T16:53:38.267Z
Comments.Text: @Veedrac I would think there are more chess people who also know about AI applied to engines than AI people that have looked at applying AI to chess. Just a hunch. And for the record, the answers touch on the highlight points about why the match wasn't fair, therefore meaning very little. It is clear the company wanted publicity rather than a fair contest :(. Unfortunately, the disinformation about A0 being wicked awesome still affects the majority of chess players until this day. Did A0 innovate? Yes, some of its tech is even in Stockfish now. Would it have beaten Stockfish fairly? No idea.
Comments.UserDisplayName: user904963
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-25T00:27:54.937Z
Comments.Text: @user904963 I made my comment with respect to the answers we actually got. I still think the topic would be better served if addressed by people with better understanding of what the technical terms mean, with the exception of the Tord Romstad quote which naturally was accurate even if the context was a little different.
Comments.UserDisplayName: Veedrac
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-25T19:46:15.983Z
Comments.Text: @Veedrac Well, it's a highly concentrated specialty to have studied machine learning and to have interest in applying it to evaluate the value of a position. I would think there would be few answers on either Stack Exchange. The guy asking this question is on the right trail though -- A0 team used jumbo specialized hardware to run their engine and a regular CPU to run Stockfish. Not fair! There were some additional odd choices like the oddly small hash size and the configuration to use "use 3 seconds per move" rather than letting Stockfish manage its own time (has a ton of code to do that).
Comments.UserDisplayName: user904963
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-26T16:40:11.227Z
Comments.Text: @user904963 I don't want to have this debate in comments, but again, I think if you asked research experts who work with the hardware under question as part of their jobs and understand why papers ablate differences not under study, they would be able to make a lot more sense of the choices here.
Comments.UserDisplayName: Veedrac
Tags: <engines><stockfish><computer-chess><hardware><alphazero>
Title: Hardware used in AlphaZero vs Stockfish match
ThreadText: # Hardware used in AlphaZero vs Stockfish match
  
  (Asked by: username_0 on 2017-12-08)
  
  I understand that AlphaZero has to use a different kind of hardware than regular Stockfish. I would expect that the hardware has a large effect on engine strength. That's why I wonder whether there have been any attempts made to provide comparable hardware to both. Also what would "comparable" mean here?
  
  Specifically I read that people complain about:
  
  * Stockfish being given only 1 GB of cache, and
  * the time limit of 1 min/move (How would this disadvantage Stockfish?)
  
  **Tags:** engines, stockfish, computer-chess, hardware, alphazero
  
  > **Comments:**
  > **username_1**: *Hash* size, not cache, apparently.  
  > **username_2**: I *strongly* suggest asking this on a technical forum (like the [AI Stack Exchange](https://ai.stackexchange.com/), perhaps titled "Fairness of evaluation in AlphaZero vs. Stockfish matches"), because the answers here are not good.  
  > **username_3**: @username_2 I would think there are more chess people who also know about AI applied to engines than AI people that have looked at applying AI to chess. Just a hunch. And for the record, the answers touch on the highlight points about why the match wasn't fair, therefore meaning very little. It is clear the company wanted publicity rather than a fair contest :(. Unfortunately, the disinformation about A0 being wicked awesome still affects the majority of chess players until this day. Did A0 innovate? Yes, some of its tech is even in Stockfish now. Would it have beaten Stockfish fairly? No idea.  
  > **username_2**: @username_3 I made my comment with respect to the answers we actually got. I still think the topic would be better served if addressed by people with better understanding of what the technical terms mean, with the exception of the Tord Romstad quote which naturally was accurate even if the context was a little different.  
  > **username_3**: @username_2 Well, it's a highly concentrated specialty to have studied machine learning and to have interest in applying it to evaluate the value of a position. I would think there would be few answers on either Stack Exchange. The guy asking this question is on the right trail though -- A0 team used jumbo specialized hardware to run their engine and a regular CPU to run Stockfish. Not fair! There were some additional odd choices like the oddly small hash size and the configuration to use "use 3 seconds per move" rather than letting Stockfish manage its own time (has a ton of code to do that).  
  > **username_2**: @username_3 I don't want to have this debate in comments, but again, I think if you asked research experts who work with the hardware under question as part of their jobs and understand why papers ablate differences not under study, they would be able to make a lot more sense of the choices here.  
  
  ---
  
  ## 7 answers
  
  ### Answer by username_15 on 2017-12-08. Score: 25 | Accepted answer
  
  > That's why I wonder whether there have been any attempts made to
  > provide comparable hardware to both.
  
  This is Google you're talking about! So the answer is obviously "No".
  
  From [the original paper](https://arxiv.org/pdf/1712.01815.pdf) hardware used for initialising and training -
  
  > Training proceeded for 700,000 steps (mini-batches of size 4,096)
  > starting from randomly initialised parameters, using 5,000
  > first-generation TPUs (15) to generate self-play games and 64
  > second-generation TPUs to train the neural networks
  
  and hardware used for the games -
  
  > AlphaZero and the previous AlphaGo Zero used a single machine with 4
  > TPUs Stockfish and Elmo played at their strongest skill level using 64
  > threads and a hash size of 1GB.
  
  So, AlphaZero used special hardware developed by Google. It used specialized Tensor Processor Units (TPUs) rather than general Central Processing Units (CPUs) as are available commercially.
  
  This is how Wikipedia describes the second generation TPUs they used -
  
  > The second generation TPU was announced in May 2017. Google stated the
  > first generation TPU design was memory bandwidth limited, and using 16
  > GB of High Bandwidth Memory in the second generation design increased
  > bandwidth to 600 GB/s and performance to 45 TFLOPS. The TPUs are then
  > arranged into 4-chip 180 TFLOPS modules
  
  They used 4 TPUs for the games, so a processing power of 180 TFLOPS.
  Note TFLOPS = 1000 billion floating point operations per second.
  
  For comparison Intel's latest most powerful chip is the Core i9 Extreme Edition processor which clocks in at 1 TFLOP. A top of the line I7 that you would find in a gaming machine would typically be about 100 GFLOPs (i.e. one tenth of a TFLOP).
  
  I think it's fair to say that AlphaZero was using an 800 pound gorilla of a hardware configuration compared to Stockfishes mouse.
  
  > **Comments:**
  > **username_16**: FLOPS stands for floating point operations per second. Floating point arithmetic is probably not used at all in the core algorithmes of Stockfish and AlphaZero. So the number of FLOPS is not really a meaningfull measure of processor speed relevant to the chess engine.  
  > **username_1**: On the contrary, I believe neural nets use floating point arithmetic quite intensively. (But of course your remark makes perfect sense and applies to Stockfish.)  
  > **username_17**: A far more apt comparison would be with a GPU; the NVIDIA Tesla V100 can do 120 TFLOPS, up from the previous generation (P100) which only did about 20. CPUs are optimized very differently from GPUs that do sheer volumes of numbers.  
  
  ---
  
  ### Answer by username_4 on 2017-12-09. Score: 15
  
  I think it's best if I elaborate on your second point with an example move in the game 1 between AlphaZero and Stockfish which also served to satisfy my curiosity today.
  
  > the time limit of 1 min/move (How would this disadvantage Stockfish?)
  
  Stockfish's performance is dependent upon both the time limit *and* the hardware configuration, so just think of when someone doubles the CPU threads, then Stockfish needs less time (not necessarily half) to find the solution than it would with the first configuration.
  
  [On the first report that was posted on Chess.com](https://www.chess.com/news/view/google-s-alphazero-destroys-stockfish-in-100-game-match) someone claimed that Stockfish was not playing optimally because he couldn't reproduce the same results using the same Stockfish on his computer. He said that on the position below (game 1 - move 11) Stockfish played Kg1-h1 (moved its king) which made no sense at all. On the other hand, stockfish on his computer showed a more developing move like Be3 (move the dark square bishop), lets look at the position:
  
  [![Game 1 on move 11](https://raw.githubusercontent.com/abdolrezat/Stockfish/Figures/figures/move11.png)](https://raw.githubusercontent.com/abdolrezat/Stockfish/Figures/figures/move11.png)
  
  Yes, it was a passive move and it seems that Stockfish should have played a more developing move. But he was wrong. Why? Because he ran Stockfish for 15 seconds, and if he had run it for an hour he would have gotten Kg1-h1 as the best move in that position. Stockfish changes it's decision when it analyses all the possible moves in more depth. Here's [what I originally said in my reply](https://www.chess.com/news/view/google-s-alphazero-destroys-stockfish-in-100-game-match):
  
  I ran the latest stockfish on the position (at move 11):
  
  * At first, It gives b4 as the optimal move when the engine is running for about a minute. After that, it decides Be3 is better.
  * But after 5 minutes on my hardware that runs on 1,400k nodes/s it will decide to go with Kh1 as the optimal move.
  * In the paper, it is said that stockfish calculates 70,000k positions per second and is run for 1 minute per move, that's about 50 times my hardware, so I'll let mine run for 50 minutes... Kg1-h1 is still the choice for Stockfish.
  
  ## Time limit is the key
  
  In the above case, it probably didn't matter much if Stockfish ran for twice the time because the decision would have been the same, but on the next move it **definitely would**:
  
  [![enter image description here](https://raw.githubusercontent.com/abdolrezat/Stockfish/Figures/figures/move12.png)](https://raw.githubusercontent.com/abdolrezat/Stockfish/Figures/figures/move12.png)
  
  In this position, Stockfish chose to move the pawn on the left side (**a4-a5**). Let's say I have a computer that runs the Stockfish engine at a speed of 1,400k nodes per second, that's about 50 times lower than the Stockfish in the real game ([In the paper](https://arxiv.org/pdf/1712.01815.pdf), it says 70,000k n/s). So I can simulate the game if I run it for 50 minutes at each move. Okay.
  
  I ran Stockfish analysis on the above position and I got the following results:
  
  * Stockfish started out suggesting some moves, but after 6 minutes on my computer (corresponds to **7.2 seconds** on the Stockfish in the real game) **it preferred a4-a5 just as the game went**.
  
  That's good, but I kept it running for a complete 50 minutes in order to reach the computations of the Stockfish in the game that was allowed 1 minute:
  
  * After 18 minutes the line changed back to **Qc3**.
  * Again, after 24 minutes the best move was **Ne3**.
  * After 27 minutes, it suggested **Ng1** all the way up to 56 minutes. After that, the optimal move was **Nf3xe5** but it's past the 50 minutes mark I mentioned before ([you can see the engine analysis for an hour and 22 minutes in this screenshot](https://raw.githubusercontent.com/abdolrezat/Stockfish/Figures/figures/120engineanalysis_LI.jpg)).
  
  The sad truth is that I believe Stockfish lost all its games because of the time limit. Stockfish gets a more in-depth search and evaluation as the time passes and in the game it wasn't allowed to use an opening book which makes it consider many moves in shallow depths. Note that in the actual game *a4-a5* was played which shows that (assuming it could evaluate 70 million positions per second) the Stockfish in the game didn't spend more than 21.6 seconds on the move. Otherwise, it would have changed its decision to those three other moves in the actual game. The reason for this is still unclear to me since my Stockfish was also consuming less memory (about ~130MB of RAM compared to the 1GB mentioned [in the original paper](https://arxiv.org/pdf/1712.01815.pdf), assuming all of it goes to hash tables).
  
  ---
  
  ## Conclusion
  
  The hardware that ran Stockfish, as I pointed out, was at the very best 18 times faster than mine (Update: on a single core) based on the move I analyzed. I'm not sure if AlphaZero could really make use of such hardware to train its networks in 4 hours, I can only assume it's too low for a game like chess. Besides, AlphaZero spent those hours on learning which also includes building solid openings (and as the paper points out, preferences over certain openings). On the other hand, Stockfish was handicapped on openings, and it did not evaluate 70 million positions per second for 60 seconds on each move.
  
  As a final note, all the things I said were based on my assumptions. Of course, the outcome of AlphaZero and the games were super interesting to me. However, I would have loved to see a game where the Stockfish play was just like what I get on my computer, too. That is, more time and an opening book allowed. It's also easy to get the outputs of Stockfish analysis on every move, and I wish they release it in order to show how well it performed.
  
  > **Comments:**
  > **username_5**: Regarding time limit, Figure 2 in AlphaZero paper shows opposite: Stockfish is better in lower budget, but scales worse, when more power is available. https://arxiv.org/pdf/1712.01815.pdf  
  > **username_4**: @username_5 Thanks for pointing that out. As I said, the performance of Stockfish (and AlphaZero) is dependent on the hardware as well as the time limit. For Instance, if we gave Stockfish more hardware (and AlphaZero less) and regenerated that figure, its line could be transformed such that it always stays above the AlphaZero line. So I guess that's a good point in comparing the hardware for both systems which also answers the main question.  
  
  ---
  
  ### Answer by username_6 on 2017-12-10. Score: 9
  
  One of the original authors of Stockfish answers the specific complaints you mentioned here:
  
  > Meanwhile Chess.com also received a lengthy comment from one of the
  > original Stockfish authors, Tord Romstad, which we'll give in full:
  >
  > > The match results by themselves are not particularly meaningful
  > > because of the rather strange choice of time controls and Stockfish
  > > parameter settings: The games were played at a fixed time of 1
  > > minute/move, which means that Stockfish has no use of its time
  > > management heuristics (lot of effort has been put into making
  > > Stockfish identify critical points in the game and decide when to
  > > spend some extra time on a move; at a fixed time per move, the
  > > strength will suffer significantly). The version of Stockfish used is
  > > one year old, was playing with far more search threads than has ever
  > > received any significant amount of testing, and had way too small hash
  > > tables for the number of threads. I believe the percentage of draws
  > > would have been much higher in a match with more normal conditions.
  > >
  > > On the other hand, there is no doubt that AlphaZero could have played
  > > better if more work had been put into the project (although the "4
  > > hours of learning" mentioned in the paper is highly misleading when
  > > you take into account the massive hardware resources used during those
  > > 4 hours). But in any case, Stockfish vs AlphaZero is very much a
  > > comparison of apples to orangutans. One is a conventional chess
  > > program running on ordinary computers, the other uses fundamentally
  > > different techniques and is running on custom designed hardware that
  > > is not available for purchase (and would be way out of the budget of
  > > ordinary users if it were).
  > >
  > > From another perspective, the apples vs orangutans angle is the most
  > > exciting thing about this: We now have two extremely different (both
  > > on the hardware and the software side) man-made entities that both
  > > display super-human chess playing abilities. That's much more
  > > interesting than yet another chess program that does the same thing as
  > > existing chess programs, just a little better. Furthermore, the
  > > adaptability of the AlphaZero approach to new domains opens exciting
  > > possibilities for the future.
  > >
  > > For chess players using computer chess programs as a tool, this
  > > breakthrough is unlikely to have a great impact, at least in the short
  > > term, because of the lack of suitable hardware for affordable prices.
  > >
  > > For chess engine programmers -- and for programmers in many other
  > > interesting domains -- the emergence of machine learning techniques
  > > that require massive hardware resources in order to be effective is a
  > > little disheartening. In a few years, it is quite possible that an
  > > AlphaZero like chess program can be made to run on ordinary computers,
  > > but the hardware resources required to *create* them will still be way
  > > beyond the budget of hobbyists or average sized companies. It is
  > > possible that an open source project with a large distributed network
  > > of computers run by volunteers could work, but the days of hundreds of
  > > unique chess engines, each with their own individual quirks and
  > > personalities, will be gone.
  
  Source: <https://www.chess.com/news/view/alphazero-reactions-from-top-gms-stockfish-author>
  
  
  ---
  
  ### Answer by username_7 on 2017-12-18. Score: 6
  
  Running on comparable hardware would be required if Google's end goal was to build a superior chess engine, but this exercise wasn't really about chess. Chess is just a convenient way to demonstrate the AI's ability to learn complicated tasks from scratch. If it can perform well against some vaguely reasonable configuration of Stockfish, it's checked the box.
  
  I predict the Google team will not spend much more effort on chess; instead, they will move on to other problems that AI has never been able to accomplish.
  
  > **Comments:**
  > **username_8**: I've given +1 because I have the same feeling.  
  > **username_0**: Sounds probable, though I doubt they would have published it, if Alphazero had lost by only a small margin (meaning that it would still be of comparable strength to stockfish).  
  > **username_7**: @username_0 They probably ran it lots of times to figure out the minimum amount of learning time it needed to crush Stockfish. Then they did a final run and published those results.  
  
  ---
  
  ### Answer by username_9 on 2017-12-09. Score: 3
  
  Visit Talkchess Forum to know more, there is where you will find some 3000 programmers.
  This was all a scam.
  Alpha played on 30 times bigger hardware than SF, 4TPUs vs 64 cores.
  4TPUs is around 1000 cores or even more.
  Alpha had simulated opening book, trained on countless top GM winning games.
  SF had very little hash.
  TC was fixed at 1 minute per move, which is again detrimental to SF, which has advanced time management.
  TPUs lack the SMP inefficiencies with more cores, so the hardware advantage was even bigger.
  Etc, etc., so basically, this was just a huge publicity stunt on the part of Google.
  Currently, Alpha is around 2800 on single core, so 400 elos below SF, and will not advanced much in the future, as, from now on, it will need advanced evaluation it will not be able to discover.
  Concerning the 4-hours issue, well, LOL, this was 48 hours ago, so now Alpha is at 5000 elo?
  Come on.
  
  > **Comments:**
  > **username_10**: You seem to believe that AlphaZero does the same thing as Stockfish, only 1000 times faster because it used 1000 times stronger hardware. This is not true at all. It uses a very different approach and that approach is very resource intensive. In fact, during the match AlphaZero was evaluating 80 thousand positions per second while Stockfish was clocking at 70 million positions per second. Now tell me that AlphaZero won because of a stronger hardware. Of course on 64 CPU it would be slower and who knows how it would play but the point is that AlphaZero does it better, albeit at higher cost.  
  > **username_9**: SF hardware costs less than $10k, Alpha one more than $250k. Draw the conclusions yourself. Nps are meaningless, and every chess programmer knows that. You can do all kinds of tricks so that nps get lower, but that does not mean you will play stronger. I would like to see it implement that approach on SF hardware and SF its on Alpha hardware. Guess the result? +85 -0 =15 for SF. If they are so great, let them publish their code.  
  > **username_11**: "Alpha had simulated opening book, trained on countless top GM winning games." Simulated opening book, yes, but it trained on GM games? Do you have a source for this? My understanding was that Alpha was entirely bootstrapped.  
  
  ---
  
  ### Answer by username_12 on 2018-09-14. Score: 0
  
  Stockfish is constrained to CPUs so it will never be able to scale to the level that GPUs are able.
  
  Gor matrix calculations GPUs scale with the n, while CPUs scale with n3, these tensor cores are further optimized so it's likely even better performance as you scale.
  
  
  ---
  
  ### Answer by username_13 on 2017-12-12. Score: -2
  
  First paragraph more detail, second short and simple answer third paragraph my opinions on the situation
  
  With AlphaZero the hardware has 0 effect on the strength of its play. It may take longer but not because it’s thinking. It’s a neural network, which means you feed it info in a vector (a single column table) it does simple math thru a giant tensor (a 3 or more dimensional table) then it spits out the answer.
  Stockfish needs time to be good because it checks possible positions to see if a move is beneficial, so the longer it looks at the problem the more position/moves it can check.
  
  There isn’t really a comparable hardware setup. Because they have different needs, Stockfish needs to analyze more positions while AlphaZero just needs to make a move. And people are upset because AlphaZero's computer is technically much more powerful and they think they should be equal in that regard. But, AlphaZero doesn’t need that supercomputer after training.
  
  In my opinion it doesn’t matter what they give to either side, unless Stockfish has an unreasonable amount of time it will probably tie a few more games but in general a similar effect will happen. This is why I think this, Stockfish initially evaluates with pieces and their values while, Alpha played (probably) millions of games to realize what is important strategically. Which is why Alpha sacs a lot more than Stockfish would ever, but gains huge positional advantages.
  
  > **Comments:**
  > **username_14**: That's just wrong. AlphaZero does tree search. More hardware makes it stronger. And it needs a ton of hardware to play better than stockfish.  
  > **username_8**: Nothing in this post is correct...  
  > **username_13**: This system is based on 3 neuralnets and partial Monroe Carlo tree search, so you are correct about it using tree search. During play it uses 2 neural network principles developed in alphago of value and policy. Go is a game that can’t be done via computational power because it’s astronomically more complex than chess. So if I believe that it needs more computing power than a brute force algorithm then you are delusional. Or miss informed. The third neural net is used to try and excelerate the learning process by guessing the back propagation changes. Alphazeros power is in the nets not mcts.  
  
  ---
  
  