Answers.Body: <p>I started doing something similar some time ago. So I can give you some advice. As already stated by @Student T, you need to define precisely which is the question you want to answer with statistics. I will not repeat his argument which is complete enough, but let me just say that defining well your problem is essential to select your methods of analysis and the selection of data.</p>
  
  <ol>
  <li><p>The dataset. If you are interested in analysing games, one good option might be the <a href="http://www.theweekinchess.com/twic" rel="nofollow noreferrer">The Week In Chess database</a>, which includes all the games of international competitions (= games of high-rated players). Another possibility is the <a href="http://www.top-5000.nl/pgn.htm" rel="nofollow noreferrer">million base</a>, containing 2.2 million games. Whereas if you are interested in analysing ratings, you can download FIDE rating list <a href="http://ratings.fide.com/download.phtml" rel="nofollow noreferrer">here</a>, for example. In downloading the dataset(s) you want to use, you should consider the following:</p>
  
  <p>1.1. The format: chessbase format is proprietary, which means it is difficult to know which are its specifications and, in consequence, to extract the data. The format pgn is more convenient, you can process it as a textfile and extract the information you need using the <a href="http://www.saremba.de/chessgml/standards/pgn/pgn-complete.htm" rel="nofollow noreferrer">pgn file specifications</a>.</p>
  
  <p>1.2. How to filter the data / which data to include in the database (in statistical parlance, sampling and stratification issues). For this you need to specify which is your population. Are you interested in games of high-rated players? Then, it is easy to find the games and databases include many. In contrast, it is more difficult to find games of unrated or lower-rated players, but you can find some of them in the websites of some national chess federations (<a href="http://chess.ca/pgn-downloads" rel="nofollow noreferrer">Canada</a>, <a href="http://www.chessscotland.com/gamedownload/games.htm" rel="nofollow noreferrer">Scotland</a>, <a href="http://www.welshchessunion.org.uk/games" rel="nofollow noreferrer">Wales</a>, <a href="http://www.icu.ie/downloads?utf8=%E2%9C%93&amp;type=pgn&amp;description=&amp;year=&amp;access=all&amp;order=updated&amp;button=" rel="nofollow noreferrer">Ireland</a>...). Also, there might be differences in the behaviour of chess players depending on the type of competition (individual or by teams; human, computer or correspondence games; etc.). Do you include all type of competitions or you leave out some of them?</p>
  
  <p>1.3. Rating systems inhomogeneities. Chess rating systems are usually based on professor Arpad <a href="http://en.wikipedia.org/wiki/Elo_rating_system" rel="nofollow noreferrer">Elo rating system</a>, but with different coefficient values. For example, USCF and FIDE ratings are not equivalent. And the value of coefficients can also change with time (see <a href="http://www.fide.com/component/content/article/1-fide-news/5421-changes-to-rating-regulations.html" rel="nofollow noreferrer">here</a> for an example).</p></li>
  <li><p>Some bibliography. I can hint you little reference material. Some time ago I did a quick search on Scopus and didn't find any interesting article (but you could try to dig deeper). However, some years ago I looked through a book on <a href="http://books.google.fr/books/about/Schach_und_Zahl.html?id=cR0QmgEACAAJ&amp;redir_esc=y" rel="nofollow noreferrer">chess and mathematics</a>. It dealt with geometrical, combinatorial and probability issues, but not with statistical analyses. It was originally written in German and there is also a translation to Spanish, but I don't think there is an English version. You can find some similar references in <a href="http://mathworld.wolfram.com/Chess.html" rel="nofollow noreferrer">Wolfram MathWorld's Chess article</a>. You might also be interested in <a href="https://chess.stackexchange.com/questions/982/do-strong-players-move-their-pieces-with-different-relative-frequencies-than-wea/1117#1117">this analysis</a> of piece moving rate as a function of Elo rating. Regarding the rating system, the seminal work is professor Elo's book <em><a href="http://books.google.fr/books/about/The_rating_of_chessplayers_past_and_pres.html?id=8pMnAQAAMAAJ&amp;redir_esc=y" rel="nofollow noreferrer">The rating of chessplayers, past and present</a></em>.</p></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2014-12-08T18:52:28.233Z
Answers.Text: thanks for your answer, I've updated my question -- I'm basically looking to develop a toolkit which will allow me to tinker around for recreational purposes -- have updated the questions
Answers.UserDisplayName: user1172468
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2014-12-08T18:53:58.177Z
Answers.Text: Another comment I'm looking for existing chess databases that are open source -- so if that is an important part of my question
Answers.UserDisplayName: user1172468
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2014-12-08T23:26:36.237Z
Answers.Text: I have added some resources: a link to the million base database, links to some national federations pgn files, and a statistical analysis answering a question in chess stackexchange. If you want to tinker around, the 2,2 million games of the million base should be more than enough. On the other side, there is no copyright for chess games. It's the database format that is propietary or not. You can open the pgn files with a text editor to see how they are made. And it is easy (although somewhat tricky) to import the data into R (using the command readLines, for example).
Answers.UserDisplayName: lodebari
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2014-12-07T00:11:22.307Z
Answers.LastActivityDate: 2014-12-08T23:16:32.897Z
Answers.LastEditDate: 2017-04-13T12:48:53.193Z
Answers.UserDisplayName: lodebari
Answers.Body: <p>I'll leave others to point you a correct database for the games because I'll focus more on the statistical part.</p>
  
  <p>First, you need to ask yourself what you want to analyze. If you don't know, you should really ask yourself what problem you want to address. Your question simply say "I want to do something, but I don't know what to do.". There're infinite ways how a database of chess games can be analyzed. I'll point to you for some areas.</p>
  
  <ol>
  <li>Whether white has an advantage</li>
  </ol>
  
  <p>Given a large data set, if we assume the games are independent we might strip away the draws. Then we might assume a binomial distribution with p=0.5 and try to detect statistical significance. However, this approach assumes that the players are equal.</p>
  
  <ol start="2">
  <li>There's rumor that Carlsen got most of his points from players older than him (last generation like Anand) but suffers against like Caruana and Giri. Is that true?</li>
  </ol>
  
  <p>You might use a two samples-mean t-test or something like that.</p>
  
  <ol start="3">
  <li><p>Whether Elo rating is really inflated</p></li>
  <li><p>Given a set of games between two players (or computers), is one player really better than another?</p></li>
  </ol>
  
  <p>For example, Carlsen has beaten Nakamura many games without a loss. Does it mean Carlsen is a better player? Did that happen because Carlsen was simply lucky.</p>
  
  <ol start="5">
  <li>Correlation between world-number one and world champion</li>
  </ol>
  
  <p>Is there a positive correlation? Are they independent?</p>
  
  <ol start="6">
  <li>Whether the Elo rating system really make sense</li>
  </ol>
  
  <p>There were some articles on Chessbase about it a while ago. I don't remember the names so you'll need google yourself.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2014-12-08T18:53:10.013Z
Answers.Text: thanks for your answer, I've updated my question -- I'm basically looking to develop a toolkit which will allow me to tinker around for recreational purposes -- have updated the questions
Answers.UserDisplayName: user1172468
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2014-12-01T23:47:10.537Z
Answers.LastActivityDate: 2014-12-02T09:03:56.057Z
Answers.LastEditDate: 2014-12-02T09:03:56.057Z
Answers.UserDisplayName: SmallChess
Answers.Body: <p>That's an unique thing that you are doing. However to help you, to collect most of all the games played by recognized rated players around the world, the below website will help you.</p>
  
  <p><a href="http://database.chessbase.com/" rel="nofollow">database.chessbase.com/</a></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2014-12-08T18:52:47.307Z
Answers.Text: @Paven Nadig thanks for your answer, I've updated my question -- I'm basically looking to develop a toolkit which will allow me to tinker around for recreational purposes -- have updated the questions
Answers.UserDisplayName: user1172468
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2014-12-08T18:54:31.507Z
Answers.Text: this is a commercial database no? I'm looking for open source databases. Thanks.
Answers.UserDisplayName: user1172468
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2014-12-01T21:50:53.967Z
Answers.LastActivityDate: 2014-12-01T21:50:53.967Z
Answers.UserDisplayName: Pavan Nadig
Body: <p>I am a graduate student studying statistics and I play chess once in a while for fun.</p>
  
  <p>I was thinking (just for the fun of it) -- applying some of what I've learned in statistics to chess games: I want to perform statistical analysis of chess games.</p>
  
  <p>Note: My primary purpose is recreation:</p>
  
  <ol>
  <li>Tinkering around</li>
  <li>Re-producing earlier works -- for example how much of an advantage does white have over black</li>
  <li>Looking at mid-game winning patterns</li>
  <li>Etc. etc.</li>
  </ol>
  
  <p>So to that:</p>
  
  <ol>
  <li>I need a large collection of chess games (historic data)</li>
  <li>I will then analyze them using R which is a statistical programming language</li>
  </ol>
  
  <p>Question:</p>
  
  <ol>
  <li>I'm looking a collection of games that I can then use as my data set (I would love to get a list of open source chess databases -- commercial chess database might have restrictions on em)</li>
  <li>I'm looking for any reference material -- like seminal papers in this area.</li>
  </ol>
  
  <p>Thanks.</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2014-12-01T23:35:19.143Z
Comments.Text: What do you want to analyze? There're lots of statistical algorithm and lots of areas that you can do it.
Comments.UserDisplayName: SmallChess
Tags: <statistics><chess-algorithms>
Title: Chess Statistics How to?
ThreadText: # Chess Statistics How to?
  
  (Asked by: username_0 on 2014-12-01)
  
  I am a graduate student studying statistics and I play chess once in a while for fun.
  
  I was thinking (just for the fun of it) -- applying some of what I've learned in statistics to chess games: I want to perform statistical analysis of chess games.
  
  Note: My primary purpose is recreation:
  
  1. Tinkering around
  2. Re-producing earlier works -- for example how much of an advantage does white have over black
  3. Looking at mid-game winning patterns
  4. Etc. etc.
  
  So to that:
  
  1. I need a large collection of chess games (historic data)
  2. I will then analyze them using R which is a statistical programming language
  
  Question:
  
  1. I'm looking a collection of games that I can then use as my data set (I would love to get a list of open source chess databases -- commercial chess database might have restrictions on em)
  2. I'm looking for any reference material -- like seminal papers in this area.
  
  Thanks.
  
  **Tags:** statistics, chess-algorithms
  
  > **Comments:**
  > **username_1**: What do you want to analyze? There're lots of statistical algorithm and lots of areas that you can do it.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2014-12-07. Score: 2 | Accepted answer
  
  I started doing something similar some time ago. So I can give you some advice. As already stated by @Student T, you need to define precisely which is the question you want to answer with statistics. I will not repeat his argument which is complete enough, but let me just say that defining well your problem is essential to select your methods of analysis and the selection of data.
  
  1. The dataset. If you are interested in analysing games, one good option might be the [The Week In Chess database](http://www.theweekinchess.com/twic), which includes all the games of international competitions (= games of high-rated players). Another possibility is the [million base](http://www.top-5000.nl/pgn.htm), containing 2.2 million games. Whereas if you are interested in analysing ratings, you can download FIDE rating list [here](http://ratings.fide.com/download.phtml), for example. In downloading the dataset(s) you want to use, you should consider the following:
  
     1.1. The format: chessbase format is proprietary, which means it is difficult to know which are its specifications and, in consequence, to extract the data. The format pgn is more convenient, you can process it as a textfile and extract the information you need using the [pgn file specifications](http://www.saremba.de/chessgml/standards/pgn/pgn-complete.htm).
  
     1.2. How to filter the data / which data to include in the database (in statistical parlance, sampling and stratification issues). For this you need to specify which is your population. Are you interested in games of high-rated players? Then, it is easy to find the games and databases include many. In contrast, it is more difficult to find games of unrated or lower-rated players, but you can find some of them in the websites of some national chess federations ([Canada](http://chess.ca/pgn-downloads), [Scotland](http://www.chessscotland.com/gamedownload/games.htm), [Wales](http://www.welshchessunion.org.uk/games), [Ireland](http://www.icu.ie/downloads?utf8=%E2%9C%93&type=pgn&description=&year=&access=all&order=updated&button=)...). Also, there might be differences in the behaviour of chess players depending on the type of competition (individual or by teams; human, computer or correspondence games; etc.). Do you include all type of competitions or you leave out some of them?
  
     1.3. Rating systems inhomogeneities. Chess rating systems are usually based on professor Arpad [Elo rating system](http://en.wikipedia.org/wiki/Elo_rating_system), but with different coefficient values. For example, USCF and FIDE ratings are not equivalent. And the value of coefficients can also change with time (see [here](http://www.fide.com/component/content/article/1-fide-news/5421-changes-to-rating-regulations.html) for an example).
  2. Some bibliography. I can hint you little reference material. Some time ago I did a quick search on Scopus and didn't find any interesting article (but you could try to dig deeper). However, some years ago I looked through a book on [chess and mathematics](http://books.google.fr/books/about/Schach_und_Zahl.html?id=cR0QmgEACAAJ&redir_esc=y). It dealt with geometrical, combinatorial and probability issues, but not with statistical analyses. It was originally written in German and there is also a translation to Spanish, but I don't think there is an English version. You can find some similar references in [Wolfram MathWorld's Chess article](http://mathworld.wolfram.com/Chess.html). You might also be interested in [this analysis](https://chess.stackexchange.com/questions/982/do-strong-players-move-their-pieces-with-different-relative-frequencies-than-wea/1117#1117) of piece moving rate as a function of Elo rating. Regarding the rating system, the seminal work is professor Elo's book *[The rating of chessplayers, past and present](http://books.google.fr/books/about/The_rating_of_chessplayers_past_and_pres.html?id=8pMnAQAAMAAJ&redir_esc=y)*.
  
  > **Comments:**
  > **username_0**: thanks for your answer, I've updated my question -- I'm basically looking to develop a toolkit which will allow me to tinker around for recreational purposes -- have updated the questions  
  > **username_0**: Another comment I'm looking for existing chess databases that are open source -- so if that is an important part of my question  
  > **username_2**: I have added some resources: a link to the million base database, links to some national federations pgn files, and a statistical analysis answering a question in chess stackexchange. If you want to tinker around, the 2,2 million games of the million base should be more than enough. On the other side, there is no copyright for chess games. It's the database format that is propietary or not. You can open the pgn files with a text editor to see how they are made. And it is easy (although somewhat tricky) to import the data into R (using the command readLines, for example).  
  
  ---
  
  ### Answer by username_1 on 2014-12-01. Score: 5
  
  I'll leave others to point you a correct database for the games because I'll focus more on the statistical part.
  
  First, you need to ask yourself what you want to analyze. If you don't know, you should really ask yourself what problem you want to address. Your question simply say "I want to do something, but I don't know what to do.". There're infinite ways how a database of chess games can be analyzed. I'll point to you for some areas.
  
  1. Whether white has an advantage
  
  Given a large data set, if we assume the games are independent we might strip away the draws. Then we might assume a binomial distribution with p=0.5 and try to detect statistical significance. However, this approach assumes that the players are equal.
  
  2. There's rumor that Carlsen got most of his points from players older than him (last generation like Anand) but suffers against like Caruana and Giri. Is that true?
  
  You might use a two samples-mean t-test or something like that.
  
  3. Whether Elo rating is really inflated
  4. Given a set of games between two players (or computers), is one player really better than another?
  
  For example, Carlsen has beaten Nakamura many games without a loss. Does it mean Carlsen is a better player? Did that happen because Carlsen was simply lucky.
  
  5. Correlation between world-number one and world champion
  
  Is there a positive correlation? Are they independent?
  
  6. Whether the Elo rating system really make sense
  
  There were some articles on Chessbase about it a while ago. I don't remember the names so you'll need google yourself.
  
  > **Comments:**
  > **username_0**: thanks for your answer, I've updated my question -- I'm basically looking to develop a toolkit which will allow me to tinker around for recreational purposes -- have updated the questions  
  
  ---
  
  ### Answer by username_3 on 2014-12-01. Score: 2
  
  That's an unique thing that you are doing. However to help you, to collect most of all the games played by recognized rated players around the world, the below website will help you.
  
  [database.chessbase.com/](http://database.chessbase.com/)
  
  > **Comments:**
  > **username_0**: @Paven username_3 thanks for your answer, I've updated my question -- I'm basically looking to develop a toolkit which will allow me to tinker around for recreational purposes -- have updated the questions  
  > **username_0**: this is a commercial database no? I'm looking for open source databases. Thanks.  
  
  ---
  
  