Answers.Body: <p>You absolutely should, which is why the two most important computer chess tournaments (<a href="https://www.chess.com/computer-chess-championship" rel="nofollow noreferrer">CCC</a> and <a href="https://tcec-chess.com/#/" rel="nofollow noreferrer">TCEC</a>) both make all competitors play on the "same" hardware. "Same" in inverted commas because the recent advent of neural-network engines like AlphaZero, which run best on GPUs, meant that it's no longer fair to play on exactly the same hardware for everyone.</p>
  
  <p>You can see the result of not using equal hardware from the 2019 WCCC results, which Komodo won. Komodo is <a href="https://ccrl.chessdom.com/ccrl/4040/" rel="nofollow noreferrer">almost a hundred elo</a> behind Stockfish when run on equal hardware (and Stockfish doesn't play in the WCCC, because it's not a commercial engine and doesn't have full-time staff to attend the event).</p>
  
  <p>The situation right now is traditional engines run on the same CPUs, NN engines run on the same GPUs, and whenever traditional engines play against NN engines, they run on consensus fair hardware. "Fair" here is a very controversial word, because there are several different measures of what is fair that I'm glossing over. Suffice to say unless you are a hardware expert yourself, I'd just trust the admins of CCC and TCEC to use something that's fair.</p>
  
  <p><strong>Edit</strong>: to illustrate the impact of hardware, here's the list of hardware the competitors played at in the <a href="https://www.chessprogramming.org/WCCC_2019#Participants" rel="nofollow noreferrer">2019 WCCC</a>. Note Jonny ran on 1200 (!) cores, and it just barely edged out Ginkgo, which was running on 1/20 of the cores. In <a href="https://www.tcec-chess.com/archive.html?season=14&amp;div=1&amp;game=1" rel="nofollow noreferrer">TCEC Season 14</a>, the two engines played with the same hardware in Division 1, and Ginkgo comfortably finished ahead of Jonny in spite of multiple crashes (which happen because Ginkgo doesn't run well at 43 cores; they result in auto-losses).</p>
  
  <p>You can also see that these games were played is Division 1, not even Division Premier. Many of the strongest engines in the world simply don't participate in WCCC. I don't know the exact reasons why, but I'm confident hardware is a major issue for participants. In turn, this lack of participation is partly why the community don't take the WCCC very seriously.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-23T12:42:39.877Z
Answers.Text: Thank you. Well, since, as you say, hardware requirements are very different between different types of engines, using energy consumption (and, obviously time) seems to be a reasonable way of levelling the field, doesn't it? Of course, it all depends on what you want to measure, but at the end it is energy that propels the world.
Answers.UserDisplayName: d-b
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-23T18:36:18.890Z
Answers.Text: Using max energy consumption is an interesting way to aim for equivalency between GPU and CPU systems.  I don’t know how it is done now, maybe that is what is done.  With the specs I provided in my answer the energy could be calculated.
Answers.UserDisplayName: Michael West
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-23T21:41:58.390Z
Answers.Text: @d-b that's one way to do it. However, GPUs are energy-intensive so equal power consumption advantages CPU engines. Another way to do it is to have equal cost (the best GPU/CPU $5000 can buy, for example) which advantages GPU engines because at this level, CPUs are more expensive than GPUs. There's also the "Leela ratio" (https://github.com/dkappe/leela-ratio), effectively basing off the AlphaZero paper. I'm sure there are more ways to say what's "fair".
Answers.UserDisplayName: Allure
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-23T21:44:36.167Z
Answers.Text: @MichaelWest I'm not sure about CCC since I don't follow it closely enough, but TCEC uses subjective judgement. Equal energy usage really advantages CPU engines, though. I suspect CCC doesn't use it, either.
Answers.UserDisplayName: Allure
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-24T05:31:31.490Z
Answers.Text: @Allure The obvious option would be introducing a new league for GPU-driven systems vs the current 'standard'  ones.
Answers.UserDisplayName: Mast
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-24T05:43:13.073Z
Answers.Text: @Mast yeah (TCEC is currently playing a CPU-only league), but people want to know what is the "strongest" engine. Right now the strongest CPU engine is undoubtedly Stockfish, while the strongest GPU engine is most likely Lc0. Which is stronger though, Stockfish or Lc0? That's only answerable by getting them to play each other, which means we need to use "fair" hardware.
Answers.UserDisplayName: Allure
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-24T07:11:43.383Z
Answers.Text: @Allure Are you sure about that? I thought GPUs are more efficient than CPUs, simply because it is easier to make HW efficient if it is more specialized. C.f., hardware dedicated to BTC-mining. A typical GPUs might use more energy than a typical CPU in a desktop system but that might be because we in most cases have reached a level of CPU performance that in most applications no longer makes it the bottleneck, hence we don't push the CPU performance that much anymore while GPUs still might be bottlenecks in certain applications. How do you compare CPU and GPU energy consumption in a fair way?
Answers.UserDisplayName: d-b
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-24T07:58:46.540Z
Answers.Text: @d-b I'm not sure, not a hardware expert here, but Googling for it there are articles like https://www.tomshardware.com/reviews/geforce-radeon-power,2122.html. Presumably you hook up the CPU & GPU and see how much power they draw. I do know the CPUs used in these competitions are vastly more powerful than ordinary desktops, and that RTX GPUs (which are the state of the art and the NN engines get a substantial boost from them compared to the previous-generation GTX GPUs) are especially energy intensive, as well.
Answers.UserDisplayName: Allure
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-23T04:18:47.173Z
Answers.LastActivityDate: 2020-02-24T03:51:54.167Z
Answers.LastEditDate: 2020-02-24T03:51:54.167Z
Answers.UserDisplayName: Allure
Answers.Body: <p>Yes, you are right.  So computer competitions will have rules about compute to keep things level.  For example the <a href="https://www.chess.com/news/view/announcing-the-new-computer-chess-championship" rel="nofollow noreferrer">chess.com computer championship</a>  uses this specification:</p>
  
  <p>CPU Engines:</p>
  
  <ul>
  <li>CPUs: 2 x Intel Xeon Platinum 8168 @ 2.70 GHz 33 MB L3   </li>
  <li>Cores: 48 physical (96 logical)   </li>
  <li>RAM: 256GB DDR4-2666 ECC Registered RDIMM  </li>
  <li>SSD: 2x Crucial MX300 (1TB) in RAID1   </li>
  <li>OS: Windows Server 2016</li>
  </ul>
  
  <p>GPU Engines:</p>
  
  <ul>
  <li>GPU: 4x Tesla V100 (64 GB GPU memory) </li>
  <li>CPU: Intel(R) Xeon(R) CPU</li>
  <li>E5-2686 v4 @ 2.30 GHz Cores: 16 physical (32 virtual) </li>
  <li>RAM: 256 GB</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-23T00:17:55.587Z
Answers.LastActivityDate: 2020-02-23T00:17:55.587Z
Answers.UserDisplayName: Michael West
Answers.Body: <p>There are a number of reasonable things you could set as constant, which is probably why they have experts determine the rules in a fundamentally arbitrary but hopefully fair fashion. Energy is one, but some other options would be:</p>
  
  <ul>
  <li><p>Hardware cost.</p></li>
  <li><p>FLOPS (floating point operations per second). </p></li>
  </ul>
  
  <p>I suspect (although my info is limited) hardware cost and energy would help neural network neural based devices, while a flops based metric would benefit classical engines, because GPUs tend to provide great FLOPS/watt and FLOPS/$ -- metrics that CPUs sacrifice for the ability to go (relatively) quickly through branchy logic. </p>
  
  <p>An argument against using a simple metric is that it might induce behavior that doesn't advance the state of the field in an interesting direction. If you ask for performance/watt or performance/$, people start asking questions like "what about an FPGA or ASIC," and eventually it becomes a competition in brute force and low level tuning, rather than interesting new ideas. Plus people wouldn't be able tinker with the engines at home if they needed specific devices to do so. </p>
  
  <p>It is better to have experts in the field push toward interesting results, IMO. Although "energy limited on commodity hardware" might be fun. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-23T21:50:12.240Z
Answers.Text: Far better to let any computer and any software make the best TOTAL SYSTEM to play chess and see how good that is.
Answers.UserDisplayName: edwina oliver
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-23T22:58:18.757Z
Answers.Text: I'm not sure about that -- are some of the chess algorithms embarrassingly parallel? In that case it is just a race to spend more money, which seems a bit silly.
Answers.UserDisplayName: Zwuwdz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-24T00:48:34.357Z
Answers.Text: It only a race for those who chose to enter.  The whole thing is silly if you think about it.  Some people choose to spend their money time effort on it but that is their choice and they are free to do it unless the govt becomes totally dictatorial.  OTOH it is a useful adjunct to research in AI and computer design and maybe even designing software.
Answers.UserDisplayName: edwina oliver
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-24T21:01:25.530Z
Answers.Text: The usefulness to AI and computer design is what makes it not silly. The concern I was trying to bring up was: the best way to make a computer more efficient is often to strip out some generality. Thus, if we want to have a competition that has a positive impact on general AI research and computer design, imposing harsh efficiency standards is counterproductive.
Answers.UserDisplayName: Zwuwdz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-24T21:04:41.267Z
Answers.Text: you keep saying what i actually said initially.  let each person be free to build the best chess playing SYSTEM that they can without any restrictions.
Answers.UserDisplayName: edwina oliver
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-24T21:12:28.907Z
Answers.Text: The distinction I'm trying to make is that allowing unbounded costs is boring -- if a competitor can get a significant advantage by spending more money, they can do that instead of pushing interesting research. However, simple hard boundaries are also less interesting, because that promotes optimizations that reduce generality. So, the current setup of having arbitrary, general purpose hardware determined by people interested in the field is the best solution. The choice of hardware is open to debate but allowing a free-for-all isn't a good solution.
Answers.UserDisplayName: Zwuwdz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-24T21:23:56.667Z
Answers.Text: best for you.  not at all good for me.  not good for finding the BEST TOTAL SOLUTION to the problem.  you will optimize a subset of what could have been done.
Answers.UserDisplayName: edwina oliver
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-24T21:39:43.853Z
Answers.Text: I think that is a reasonable thing to call the underlying source of our disagreement.
Answers.UserDisplayName: Zwuwdz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-23T21:40:31.030Z
Answers.LastActivityDate: 2020-02-23T21:40:31.030Z
Answers.UserDisplayName: Zwuwdz
Answers.Body: <p>It really comes down to the purpose of the event.</p>
  
  <p>If it is a championship that is trying to determine who wrote the best software, and not just the strongest hardware/software combination, then you want to even up the hardware as best as possible so the software is the determining factor. The winner determines the best software.</p>
  
  <p>If it is a demonstration or exhibition match, like the AlphaZero-Stockfish match, or the Kasparov-Deep Blue match years ago, and it is all about just crushing the opponent and winning at all costs, all bets are off, and you can expect to see massive, and outrageous, hardware builds.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-23T21:54:53.663Z
Answers.Text: The AlphaZero-Stockfish preprint was heavily criticized because the hardware was so busted. They corrected the issues in the final paper. Kasparov-Deep Blue I wouldn't say is a good example, since at the time it wasn't apparent how strong computers can become. It also wasn't a computer vs. computer match.
Answers.UserDisplayName: Allure
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-24T21:27:49.523Z
Answers.Text: @PhishMaster    d-b and zwuwdz want to find the best sofware that runs on a specific PC windoze configuration which narrows the solution space way too far and ensures that inferior answers will be created.
Answers.UserDisplayName: edwina oliver
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-23T16:53:20.907Z
Answers.LastActivityDate: 2020-02-23T16:53:20.907Z
Answers.UserDisplayName: PhishMaster
Answers.Body: <p>Should we limit human players who play by how much they weigh or their IQ?</p>
  
  <p>There is no point in so called 'levelling the field' when computers play each other.</p>
  
  <p>There is much more than building the biggest computer that determines who wins.  </p>
  
  <p>Software makes the hardware happen.   And how well they came up with the software and how well they determined the algorithms to guide the software as well as practical tricks they use like having table bases or pitting two machines against each other to help them learn faster are more important than the hardware. </p>
  
  <p>Limiting energy would only mean that they would use different compuer chip technology that has nothing to do with chess. </p>
  
  <p>Your suggestion would change what is being optimized and what most people care about are how well the computers play not how much electricity they use. </p>
  
  <p>However since the real concern is the software the latest championship had these rules for software: </p>
  
  <p>The rules for the World Chess SOFTWARE  Championship state that competing programs must run on machines with identical hardware specifications. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-23T00:51:15.957Z
Answers.Text: I think you've misunderstood how relevant the hardware is, your analogy is very wrong. The IQ of human players is equivalent to the quality of the software/algorithms. The hardware/power provided is equivalent to the time on the clock. Saying you shouldn't 'level the field' when computers play each other is the same as saying you should let human players use unlimited time. That then simply becomes a competition of who has the most free time, rather than who is better at chess
Answers.UserDisplayName: Darren H
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-23T01:14:58.053Z
Answers.Text: Sorry but the hardware makes a big difference too.  But software makes the hardware happen.  Hardware is like what neurons you work with.  Software is how they get used to do something useful.   Your analogy with time is totally wrong.  Both people and computers are time limited for fairness.  The OP was about levelling the hardware which is already done since 2010.  No amount of time would make any human better than their ability.  Indeed too much time is counterproductive as one gets bored lazy sleepy and makes more errors.
Answers.UserDisplayName: edwina oliver
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-23T03:58:19.803Z
Answers.Text: This is completely wrong. There is a reason why the World Computer Chess Championship doesn't command the respect of the community, if you allow competitors to use their own hardware (+ opening book) then you aren't directly measuring chess strength. CCC and TCEC, the two most important computer chess championships, both use equal hardware for all programs (although the GPU engines recently threw a spanner into the works).
Answers.UserDisplayName: Allure
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-23T04:01:42.510Z
Answers.Text: huh?????   i quoted where they do use the same hardware by the rules.   you do not understand SYSTEMS.   it takes both hardware and software to play chess.  Why should someone who can make better hardware but only good software not be able to compete with those with poor hardware but better software.  It is the TOTAL SYSTEM that really matters.
Answers.UserDisplayName: edwina oliver
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-23T04:14:44.813Z
Answers.Text: @DarrenH Leveling the field between humans would be limiting the more talented human in some manner.  This could be less sleep or time, but unlimited time doesn't reduce or enhance talent.  This could help those who don't have good time management skills, but this isn't necessarily the one with less talent.
Answers.UserDisplayName: Mike Jones
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-23T04:15:57.013Z
Answers.Text: @edwinaoliver no, I strongly disagree. Nobody is "making" hardware for these purposes, well almost nobody, you'd need the budget of Google to even think about it. Hardware is bought off the shelf and is dependant only on what funds you have available. The really clever bit is the software and that is what is being compared. I reiterate my analogy, hardware is the time, software is the brain. Allowing people to use their own hardware is equivalent to allowing a human to choose their own time controls.
Answers.UserDisplayName: Darren H
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-23T17:52:52.633Z
Answers.Text: @DarrenH  IBM made hardware.  Others use macs and linux systems not windoze.  You can expect that if hardware would help AI then someone would design special hardware.  Especially with the ease and low cost of FPGAs now.  And who knows what the future could bring if it were not limited.
Answers.UserDisplayName: edwina oliver
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-24T05:35:24.197Z
Answers.Text: @DarrenH Arguably it's more like allowing a human to take an entire team with masters with them heavily advising them every step on the way.
Answers.UserDisplayName: Mast
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-24T09:05:15.133Z
Answers.Text: @edwinaoliver here is not really the place for going into detail about the intricacies of operating systems, but suffice to say I still strongly disagree with your statement and your response leaves me with the impression that you don't really have a strong understanding of how these things actually work
Answers.UserDisplayName: Darren H
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-24T14:22:19.580Z
Answers.Text: huh???    i said let them use any hardware they want and not cripple anybody in any manner.  you just want to handicap the person who worked harder/smarter and used better hardware.  you would rule out those who use other hardware and cant run at all on what  you  dictate should be used.  you would not find the best chess playing system with your approach.  why not also make them all use the same programming language instead of letting them pick a better faster one they already know/use?
Answers.UserDisplayName: edwina oliver
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-23T00:06:50.280Z
Answers.LastActivityDate: 2020-02-23T00:18:05.347Z
Answers.LastEditDate: 2020-02-23T00:18:05.347Z
Answers.UserDisplayName: edwina oliver
Body: <p>I just read <a href="https://chess.stackexchange.com/questions/28605/knowing-that-alphazero-beat-stockfish-8-28-wins-0-losses-and-72-draws-what">this question</a> and it made me wonder: when setting up such a game/comparison - shouldn't you try to level the computing capacity field? Or at least limit how much energy each "player" is allowed to use (and then let each team optimize within that limit). Otherwise, isn't it just a matter of building the biggest computer? </p>
  
Tags: <stockfish><alphazero>
Title: Level the field when letting computers play against each other
ThreadText: # Level the field when letting computers play against each other
  
  (Asked by: username_0 on 2020-02-23)
  
  I just read [this question](https://chess.stackexchange.com/questions/28605/knowing-that-alphazero-beat-stockfish-8-28-wins-0-losses-and-72-draws-what) and it made me wonder: when setting up such a game/comparison - shouldn't you try to level the computing capacity field? Or at least limit how much energy each "player" is allowed to use (and then let each team optimize within that limit). Otherwise, isn't it just a matter of building the biggest computer?
  
  **Tags:** stockfish, alphazero
  
  
  ---
  
  ## 5 answers
  
  ### Answer by username_1 on 2020-02-23. Score: 6
  
  You absolutely should, which is why the two most important computer chess tournaments ([CCC](https://www.chess.com/computer-chess-championship) and [TCEC](https://tcec-chess.com/#/)) both make all competitors play on the "same" hardware. "Same" in inverted commas because the recent advent of neural-network engines like AlphaZero, which run best on GPUs, meant that it's no longer fair to play on exactly the same hardware for everyone.
  
  You can see the result of not using equal hardware from the 2019 WCCC results, which Komodo won. Komodo is [almost a hundred elo](https://ccrl.chessdom.com/ccrl/4040/) behind Stockfish when run on equal hardware (and Stockfish doesn't play in the WCCC, because it's not a commercial engine and doesn't have full-time staff to attend the event).
  
  The situation right now is traditional engines run on the same CPUs, NN engines run on the same GPUs, and whenever traditional engines play against NN engines, they run on consensus fair hardware. "Fair" here is a very controversial word, because there are several different measures of what is fair that I'm glossing over. Suffice to say unless you are a hardware expert yourself, I'd just trust the admins of CCC and TCEC to use something that's fair.
  
  **Edit**: to illustrate the impact of hardware, here's the list of hardware the competitors played at in the [2019 WCCC](https://www.chessprogramming.org/WCCC_2019#Participants). Note Jonny ran on 1200 (!) cores, and it just barely edged out Ginkgo, which was running on 1/20 of the cores. In [TCEC Season 14](https://www.tcec-chess.com/archive.html?season=14&div=1&game=1), the two engines played with the same hardware in Division 1, and Ginkgo comfortably finished ahead of Jonny in spite of multiple crashes (which happen because Ginkgo doesn't run well at 43 cores; they result in auto-losses).
  
  You can also see that these games were played is Division 1, not even Division Premier. Many of the strongest engines in the world simply don't participate in WCCC. I don't know the exact reasons why, but I'm confident hardware is a major issue for participants. In turn, this lack of participation is partly why the community don't take the WCCC very seriously.
  
  > **Comments:**
  > **username_0**: Thank you. Well, since, as you say, hardware requirements are very different between different types of engines, using energy consumption (and, obviously time) seems to be a reasonable way of levelling the field, doesn't it? Of course, it all depends on what you want to measure, but at the end it is energy that propels the world.  
  > **username_2**: Using max energy consumption is an interesting way to aim for equivalency between GPU and CPU systems.  I don’t know how it is done now, maybe that is what is done.  With the specs I provided in my answer the energy could be calculated.  
  > **username_1**: @username_0 that's one way to do it. However, GPUs are energy-intensive so equal power consumption advantages CPU engines. Another way to do it is to have equal cost (the best GPU/CPU $5000 can buy, for example) which advantages GPU engines because at this level, CPUs are more expensive than GPUs. There's also the "Leela ratio" (https://github.com/dkappe/leela-ratio), effectively basing off the AlphaZero paper. I'm sure there are more ways to say what's "fair".  
  > **username_1**: @MichaelWest I'm not sure about CCC since I don't follow it closely enough, but TCEC uses subjective judgement. Equal energy usage really advantages CPU engines, though. I suspect CCC doesn't use it, either.  
  > **username_3**: @username_1 The obvious option would be introducing a new league for GPU-driven systems vs the current 'standard'  ones.  
  > **username_1**: @username_3 yeah (TCEC is currently playing a CPU-only league), but people want to know what is the "strongest" engine. Right now the strongest CPU engine is undoubtedly Stockfish, while the strongest GPU engine is most likely Lc0. Which is stronger though, Stockfish or Lc0? That's only answerable by getting them to play each other, which means we need to use "fair" hardware.  
  > **username_0**: @username_1 Are you sure about that? I thought GPUs are more efficient than CPUs, simply because it is easier to make HW efficient if it is more specialized. C.f., hardware dedicated to BTC-mining. A typical GPUs might use more energy than a typical CPU in a desktop system but that might be because we in most cases have reached a level of CPU performance that in most applications no longer makes it the bottleneck, hence we don't push the CPU performance that much anymore while GPUs still might be bottlenecks in certain applications. How do you compare CPU and GPU energy consumption in a fair way?  
  > **username_1**: @username_0 I'm not sure, not a hardware expert here, but Googling for it there are articles like https://www.tomshardware.com/reviews/geforce-radeon-power,2122.html. Presumably you hook up the CPU & GPU and see how much power they draw. I do know the CPUs used in these competitions are vastly more powerful than ordinary desktops, and that RTX GPUs (which are the state of the art and the NN engines get a substantial boost from them compared to the previous-generation GTX GPUs) are especially energy intensive, as well.  
  
  ---
  
  ### Answer by username_2 on 2020-02-23. Score: 3
  
  Yes, you are right. So computer competitions will have rules about compute to keep things level. For example the [chess.com computer championship](https://www.chess.com/news/view/announcing-the-new-computer-chess-championship) uses this specification:
  
  CPU Engines:
  
  * CPUs: 2 x Intel Xeon Platinum 8168 @ 2.70 GHz 33 MB L3
  * Cores: 48 physical (96 logical)
  * RAM: 256GB DDR4-2666 ECC Registered RDIMM
  * SSD: 2x Crucial MX300 (1TB) in RAID1
  * OS: Windows Server 2016
  
  GPU Engines:
  
  * GPU: 4x Tesla V100 (64 GB GPU memory)
  * CPU: Intel(R) Xeon(R) CPU
  * E5-2686 v4 @ 2.30 GHz Cores: 16 physical (32 virtual)
  * RAM: 256 GB
  
  
  ---
  
  ### Answer by username_4 on 2020-02-23. Score: 3
  
  There are a number of reasonable things you could set as constant, which is probably why they have experts determine the rules in a fundamentally arbitrary but hopefully fair fashion. Energy is one, but some other options would be:
  
  * Hardware cost.
  * FLOPS (floating point operations per second).
  
  I suspect (although my info is limited) hardware cost and energy would help neural network neural based devices, while a flops based metric would benefit classical engines, because GPUs tend to provide great FLOPS/watt and FLOPS/$ -- metrics that CPUs sacrifice for the ability to go (relatively) quickly through branchy logic.
  
  An argument against using a simple metric is that it might induce behavior that doesn't advance the state of the field in an interesting direction. If you ask for performance/watt or performance/$, people start asking questions like "what about an FPGA or ASIC," and eventually it becomes a competition in brute force and low level tuning, rather than interesting new ideas. Plus people wouldn't be able tinker with the engines at home if they needed specific devices to do so.
  
  It is better to have experts in the field push toward interesting results, IMO. Although "energy limited on commodity hardware" might be fun.
  
  > **Comments:**
  > **username_5**: Far better to let any computer and any software make the best TOTAL SYSTEM to play chess and see how good that is.  
  > **username_4**: I'm not sure about that -- are some of the chess algorithms embarrassingly parallel? In that case it is just a race to spend more money, which seems a bit silly.  
  > **username_5**: It only a race for those who chose to enter.  The whole thing is silly if you think about it.  Some people choose to spend their money time effort on it but that is their choice and they are free to do it unless the govt becomes totally dictatorial.  OTOH it is a useful adjunct to research in AI and computer design and maybe even designing software.  
  > **username_4**: The usefulness to AI and computer design is what makes it not silly. The concern I was trying to bring up was: the best way to make a computer more efficient is often to strip out some generality. Thus, if we want to have a competition that has a positive impact on general AI research and computer design, imposing harsh efficiency standards is counterproductive.  
  > **username_5**: you keep saying what i actually said initially.  let each person be free to build the best chess playing SYSTEM that they can without any restrictions.  
  > **username_4**: The distinction I'm trying to make is that allowing unbounded costs is boring -- if a competitor can get a significant advantage by spending more money, they can do that instead of pushing interesting research. However, simple hard boundaries are also less interesting, because that promotes optimizations that reduce generality. So, the current setup of having arbitrary, general purpose hardware determined by people interested in the field is the best solution. The choice of hardware is open to debate but allowing a free-for-all isn't a good solution.  
  > **username_5**: best for you.  not at all good for me.  not good for finding the BEST TOTAL SOLUTION to the problem.  you will optimize a subset of what could have been done.  
  > **username_4**: I think that is a reasonable thing to call the underlying source of our disagreement.  
  
  ---
  
  ### Answer by username_6 on 2020-02-23. Score: 1
  
  It really comes down to the purpose of the event.
  
  If it is a championship that is trying to determine who wrote the best software, and not just the strongest hardware/software combination, then you want to even up the hardware as best as possible so the software is the determining factor. The winner determines the best software.
  
  If it is a demonstration or exhibition match, like the AlphaZero-Stockfish match, or the Kasparov-Deep Blue match years ago, and it is all about just crushing the opponent and winning at all costs, all bets are off, and you can expect to see massive, and outrageous, hardware builds.
  
  > **Comments:**
  > **username_1**: The AlphaZero-Stockfish preprint was heavily criticized because the hardware was so busted. They corrected the issues in the final paper. Kasparov-Deep Blue I wouldn't say is a good example, since at the time it wasn't apparent how strong computers can become. It also wasn't a computer vs. computer match.  
  > **username_5**: @Phishusername_3er    username_0 and zwuwdz want to find the best sofware that runs on a specific PC windoze configuration which narrows the solution space way too far and ensures that inferior answers will be created.  
  
  ---
  
  ### Answer by username_5 on 2020-02-23. Score: -1
  
  Should we limit human players who play by how much they weigh or their IQ?
  
  There is no point in so called 'levelling the field' when computers play each other.
  
  There is much more than building the biggest computer that determines who wins.
  
  Software makes the hardware happen. And how well they came up with the software and how well they determined the algorithms to guide the software as well as practical tricks they use like having table bases or pitting two machines against each other to help them learn faster are more important than the hardware.
  
  Limiting energy would only mean that they would use different compuer chip technology that has nothing to do with chess.
  
  Your suggestion would change what is being optimized and what most people care about are how well the computers play not how much electricity they use.
  
  However since the real concern is the software the latest championship had these rules for software:
  
  The rules for the World Chess SOFTWARE Championship state that competing programs must run on machines with identical hardware specifications.
  
  > **Comments:**
  > **username_7**: I think you've misunderstood how relevant the hardware is, your analogy is very wrong. The IQ of human players is equivalent to the quality of the software/algorithms. The hardware/power provided is equivalent to the time on the clock. Saying you shouldn't 'level the field' when computers play each other is the same as saying you should let human players use unlimited time. That then simply becomes a competition of who has the most free time, rather than who is better at chess  
  > **username_5**: Sorry but the hardware makes a big difference too.  But software makes the hardware happen.  Hardware is like what neurons you work with.  Software is how they get used to do something useful.   Your analogy with time is totally wrong.  Both people and computers are time limited for fairness.  The OP was about levelling the hardware which is already done since 2010.  No amount of time would make any human better than their ability.  Indeed too much time is counterproductive as one gets bored lazy sleepy and makes more errors.  
  > **username_1**: This is completely wrong. There is a reason why the World Computer Chess Championship doesn't command the respect of the community, if you allow competitors to use their own hardware (+ opening book) then you aren't directly measuring chess strength. CCC and TCEC, the two most important computer chess championships, both use equal hardware for all programs (although the GPU engines recently threw a spanner into the works).  
  > **username_5**: huh?????   i quoted where they do use the same hardware by the rules.   you do not understand SYSTEMS.   it takes both hardware and software to play chess.  Why should someone who can make better hardware but only good software not be able to compete with those with poor hardware but better software.  It is the TOTAL SYSTEM that really matters.  
  > **username_8**: @DarrenH Leveling the field between humans would be limiting the more talented human in some manner.  This could be less sleep or time, but unlimited time doesn't reduce or enhance talent.  This could help those who don't have good time management skills, but this isn't necessarily the one with less talent.  
  > **username_7**: @edwinaoliver no, I strongly disagree. Nobody is "making" hardware for these purposes, well almost nobody, you'd need the budget of Google to even think about it. Hardware is bought off the shelf and is dependant only on what funds you have available. The really clever bit is the software and that is what is being compared. I reiterate my analogy, hardware is the time, software is the brain. Allowing people to use their own hardware is equivalent to allowing a human to choose their own time controls.  
  > **username_5**: @DarrenH  IBM made hardware.  Others use macs and linux systems not windoze.  You can expect that if hardware would help AI then someone would design special hardware.  Especially with the ease and low cost of FPGAs now.  And who knows what the future could bring if it were not limited.  
  > **username_3**: @DarrenH Arguably it's more like allowing a human to take an entire team with masters with them heavily advising them every step on the way.  
  > **username_7**: @edwinaoliver here is not really the place for going into detail about the intricacies of operating systems, but suffice to say I still strongly disagree with your statement and your response leaves me with the impression that you don't really have a strong understanding of how these things actually work  
  > **username_5**: huh???    i said let them use any hardware they want and not cripple anybody in any manner.  you just want to handicap the person who worked harder/smarter and used better hardware.  you would rule out those who use other hardware and cant run at all on what  you  dictate should be used.  you would not find the best chess playing system with your approach.  why not also make them all use the same programming language instead of letting them pick a better faster one they already know/use?  
  
  ---
  
  