Answers.Body: <p>It's 2019-12 now and 5 years after the original question.</p>
  
  <ol>
  <li><p>Does any such thing exist (webcam/video to pgn converter)? A
  prototype maybe?</p>
  
  <p>see </p>
  
  <ul>
  <li><a href="https://github.com/WolfgangFahl/play-chess-with-a-webcam/issues/19" rel="noreferrer">https://github.com/WolfgangFahl/play-chess-with-a-webcam/issues/19</a></li>
  <li><a href="https://github.com/WolfgangFahl/play-chess-with-a-webcam/issues/22" rel="noreferrer">https://github.com/WolfgangFahl/play-chess-with-a-webcam/issues/22</a>
  which are both closed by now. See <a href="http://lichess.bitplan.com" rel="noreferrer">http://lichess.bitplan.com</a> for a game successfully recorded. Join <a href="https://gitter.im/play-chess-with-a-webcam/community" rel="noreferrer">https://gitter.im/play-chess-with-a-webcam/community</a> to discuss</li>
  </ul></li>
  <li><p>Has anyone attempted such a feat and shared their research?</p>
  
  <p>Yes - so far I found some 35 research papers and some 10 open source solutions and one commercial offering</p>
  
  <p>2.1  <a href="http://wiki.bitplan.com/index.php/PlayChessWithAWebCam/Papers" rel="noreferrer">http://wiki.bitplan.com/index.php/PlayChessWithAWebCam/Papers</a> </p>
  
  <p>2.2. <a href="http://wiki.bitplan.com/index.php/PlayChessWithAWebCam#Open_Source_Projects" rel="noreferrer">http://wiki.bitplan.com/index.php/PlayChessWithAWebCam#Open_Source_Projects</a></p>
  
  <p>2.3. <a href="http://wiki.bitplan.com/index.php/PlayChessWithAWebCam#Commercial" rel="noreferrer">http://wiki.bitplan.com/index.php/PlayChessWithAWebCam#Commercial</a></p>
  
  <p><a href="http://wiki.bitplan.com/index.php/PlayChessWithAWebCam" rel="noreferrer">http://wiki.bitplan.com/index.php/PlayChessWithAWebCam</a> is another project
  aiming at the goals stated by you. The project was started 2019-10-15 and in a few hours I already found a lot of resources e.g. this Question
  Some two months later I have updated my answer (see links above)</p></li>
  <li><p>Are there obvious limitations that make this project unlikely to succeed?
  I can only answer from my own experience with a similar project I did 31 years ago see <a href="http://wiki.bitplan.com/index.php/MagneticSensorChessBoard1987" rel="noreferrer">http://wiki.bitplan.com/index.php/MagneticSensorChessBoard1987</a>.
  At that time i tried to create a chess board that would detect moves. I used magnets
  in the wooden pieces and reed contacts. The key to success was using a chess engine
  that new "the rules" to be able to concentrate on movements alone and not having
  to detect individual pieces. As outlined in an answer above promotions to non queen
  pieces might be one of the edge usecases that are difficult. Detecting the motion of
  pieces via picture differences is a well understood problem.</p>
  
  <p>Chess engines and UCI compatible GUIs are also available these days. So the problem
  is more an integration problem these days then solving the individual detail issues.</p>
  
  <p>A major obstacle is the difference between a "prototype" and a product. See e.g. 
  <a href="http://www.nandanbanerjee.com/files/ICCVR-08AUG12-011%20paper.pdf" rel="noreferrer">http://www.nandanbanerjee.com/files/ICCVR-08AUG12-011%20paper.pdf</a> </p>
  
  <p>a paper that is even older than this question but far from a product that you can buy
  at a reasonable price.</p></li>
  <li><p>Would you buy it if it existed?</p>
  
  <p>Yes. When I bought a new chess board a few days ago I was looking for one that
  could record moves and it was still around 500 EUR. As it looks the hardware cost
  will be well below 150 EUR these days including a decent chess board and pieces.</p>
  
  <p>The software might well be opensource and free (as in free beer) soon. I'll update
  this answer accordingly.</p></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-16T22:23:16.477Z
Answers.Text: Thank you, I'm looking forward to seeing your progress
Answers.UserDisplayName: Riccardo Zorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-25T06:30:12.463Z
Answers.Text: I suggest to add to the knowledge, the following website: 
  
  https://github.com/maciejczyzewski/neural-chessboard
Answers.UserDisplayName: djnavas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-25T07:07:11.387Z
Answers.Text: @djnavas - thanks for the hint i think it is already part of http://wiki.bitplan.com/index.php/PlayChessWithAWebCam/Papers
Answers.UserDisplayName: Wolfgang Fahl
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-15T19:11:13.367Z
Answers.LastActivityDate: 2019-12-28T14:32:36.753Z
Answers.LastEditDate: 2019-12-28T14:32:36.753Z
Answers.UserDisplayName: Wolfgang Fahl
Answers.Body: <p>In order,</p>
  
  <p>1) Not to my knowledge</p>
  
  <p>2) Not to my knowledge</p>
  
  <p>3) There are several things that make this sort of project non-trivial, even difficult. Not the least of which is following the piece, differentiating it from a hand gesture, for example. Also determining when the move stopped would be an interesting question for computer vision to solve. These kinds of problems have been solved in other situations, but not with 100% accuracy, and  certainly not at prices that are even close to being competitive with the DGT board.</p>
  
  <p>Then there's the question of where the camera could be positioned. Directly overhead would offer the best view of the board, making it easier to determine the square with precision. But wherever the camera gets placed, there's a distinct probability its view will be obstructed by one of the players, either leaning over the board, reaching for the clock, or simply gesticulating.</p>
  
  <p>4) Not unless it was really cheap. And I don't expect it would be. My reasoning is simple. As time controls shorten, the quality of the game lessens, to the point where it's no longer interesting to me</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2014-02-10T01:23:34.797Z
Answers.Text: You make a great point at 4: the quality is indeed poor.  Not worth the hassle of setting up a camera, much less spending a month to write the software.  I just won the playoff, but I am embarassed of how poorly I played.
Answers.UserDisplayName: Riccardo Zorn
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2014-02-10T00:11:36.577Z
Answers.LastActivityDate: 2014-02-10T00:11:36.577Z
Answers.UserDisplayName: Arlen
Answers.Body: <p>I wrote this software. It contains webcam AI for move recognition, connected to Arena GUI.</p>
  
  <p><a href="https://www.youtube.com/watch?v=APkoE1UDKj0" rel="nofollow">https://www.youtube.com/watch?v=APkoE1UDKj0</a></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-06-15T12:36:07.600Z
Answers.Text: Very impressive Marius, congratulations!
  Is the alpha available for testing already?
  Can you expose the moves through a service, for use by other software?
Answers.UserDisplayName: Riccardo Zorn
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-06-15T11:55:37.523Z
Answers.LastActivityDate: 2015-06-15T11:55:37.523Z
Answers.UserDisplayName: Marius
Answers.Body: <p>That would be a rather complex task. Maybe a reasonable first step is to look into recognizing FEN position from a single image - for which more resources exists, for example:</p>
  
  <ul>
  <li><a href="http://chessgrabber.nicolaas.net/acknowledgements" rel="nofollow noreferrer">Create a FEN from a chess diagram picture</a></li>
  <li><a href="https://stackoverflow.com/questions/11288460/recommended-pattern-recognition-technique-for-chess-board">Recommended pattern recognition technique for chess board</a></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-10-24T11:01:37.330Z
Answers.Text: recognizing a still image is actually harder than having the video leading up to it, since the latter eliminates the problem of having to be able to identify the pieces.
Answers.UserDisplayName: peter
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2014-02-10T02:21:17.947Z
Answers.Text: Excellent thank you. I read both and they are really interesting.  Chessgrabber is a recent project and is definitely more suited, I contacted the developer.
Answers.UserDisplayName: Riccardo Zorn
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2014-02-11T17:32:06.157Z
Answers.Text: I'd think that from this first step it wouldn't be too difficult to convert a video to PGN - take images every few milliseconds, analyse them, and if the position has changed in a legal way add the new position to the file.
Answers.UserDisplayName: rlms
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2014-02-12T09:55:17.187Z
Answers.Text: Update: the developer responded, it will never work for 3-d skewed images with overlapping pieces. Too bad, without that as a base, the project will never start.
Answers.UserDisplayName: Riccardo Zorn
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2014-02-09T20:27:47.133Z
Answers.LastActivityDate: 2014-02-17T19:15:48.387Z
Answers.LastEditDate: 2017-05-23T12:37:05.403Z
Answers.UserDisplayName: Tihomir Georgiev
Answers.Body: <p>I worked on the chess tracking project <strong><a href="http://iie.fing.edu.uy/investigacion/grupos/gti/timag/trabajos/2013/ajedrez/index.html" rel="nofollow noreferrer">8x8cam</a></strong> within image processing course in 2013 at FING UdelaR.</p>
  <p>Then in 2014 other students created a better version for Android named <strong><a href="http://iie.fing.edu.uy/investigacion/grupos/gti/timag/trabajos/2014/ajedrez/index.html" rel="nofollow noreferrer">chesstrack</a></strong>. Under it's web you can find videos and link to the code they used.</p>
  <p>I would love to see a finalized app of this project idea I had for years. So if there is some news about this please leave a comment.</p>
  <p>Answering your pointed questions:</p>
  <ol>
  <li><p>Does any such thing exist (webcam/video to pgn converter? A prototype maybe?</p>
  <p><br>Apart from my project <a href="http://iie.fing.edu.uy/investigacion/grupos/gti/timag/trabajos/2013/ajedrez/index.html" rel="nofollow noreferrer">8x8cam</a> and <a href="http://iie.fing.edu.uy/investigacion/grupos/gti/timag/trabajos/2014/ajedrez/index.html" rel="nofollow noreferrer">chesstrack</a>, I have recently found a project named <a href="http://blogdugas.net/blog/2015/05/18/play-chess-with-a-webcam/" rel="nofollow noreferrer">ChessCam</a>. All of these projects are Open Source and only need to be named as resource if you create a new project from them.</p>
  </li>
  <li><p>Has anyone attempted such a feat and shared their research?</p>
  <p><br>Yes. You can see such on previous question's answer.</p>
  </li>
  <li><p>Are there obvious limitations that make this project unlikely to succeed?</p>
  <p><br> The hardest part I believe is to make the software work no matter how the camera is placed and independently of the illumination. And also to work on fast paced games and work with the hand occlusions when the game is being played, in order to detect correctly the move done.</p>
  </li>
  <li><p>Would you buy it if it existed?</p>
  <p><br> Definitely yes.</p>
  </li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2016-09-09T04:32:15.063Z
Answers.LastActivityDate: 2019-10-16T20:04:21.833Z
Answers.LastEditDate: 2020-06-17T09:59:59.497Z
Answers.UserDisplayName: Cedric Zoppolo
Answers.Body: <p><strong>Edit:</strong> promoting to anything different than queen will make everything explode. It will also be very hard to support this (my program for Part 2 supports underpromotion, but storing the info of Part 1 purely by reading pixels is a nightmare).</p>
  
  <p><strong>Edit2:</strong> I just noticed you mean real board chess to PGN, so this is all useless and I will most likely delete my answer.</p>
  
  <h2>Part 1: storing the squares using pixel readers</h2>
  
  <p>It is totally possible, the steps needed would be:</p>
  
  <ol>
  <li>change the set up of the board so it highlights the last move (prefereably <em>not</em> just the border)</li>
  <li>write a program that reads pixels on given (x,y)</li>
  <li>make it read about 5+ pixels on each square (so it is impossible for the cursor/piece to cover all the pixels to be read)</li>
  <li>convert the HEX to RBG, then give a <strong>range</strong> of accepted values of what we deem to be considered close to the "last move" color</li>
  <li>save the moves on an array with a format <code>["h1-h8", "e4-e5", ...]</code>  just the initial and ending squares (order <strong>unimportant</strong>)</li>
  </ol>
  
  <p><strong>Note:</strong> from here we can't really know from which square was the initial/ending position, but there are only 2 posibilities and only 1 of them is legal.</p>
  
  <p>I have written simple pixel-based macros/bots for RPGs games in C#, I can guarantee this is not <em>that</em> difficult, given that:</p>
  
  <ol>
  <li>the cam is static (so the x,y values don't get messed up)</li>
  <li>you previously know the background color or you can set it up before playing</li>
  </ol>
  
  <p>Basically, it is best if the game is played online and you are recording directly from the screen.</p>
  
  <blockquote>
    <p>Sorry, no program avaiable for part #1</p>
  </blockquote>
  
  <hr>
  
  <h2>Part 2: exporting that list into PGN</h2>
  
  <p>To turn this list into a real PGN, you can use a <a href="https://github.com/ajax333221/Isepic-Chess" rel="nofollow noreferrer">program I wrote in JavaScript</a> that records moves, and one of the functions <code>moveCaller()</code> you pass two squares positions and it searches the <code>final_pos</code> in the legal moves of <code>initial_pos</code> and if there is a match then it calls <code>makeMove()</code>. So we can call it twice <code>e5 to e4</code> and <code>e4 to e5</code> and I guarantee it will ignore the illegal one.</p>
  
  <ol>
  <li>download the project (direct link <a href="https://github.com/ajax333221/Isepic-Chess/archive/master.zip" rel="nofollow noreferrer">.zip</a>)</li>
  <li>uncompress the <code>.zip</code></li>
  <li>change the JavaScript of the <code>index.html</code> file with this:</li>
  </ol>
  
  <p>'</p>
  
  <pre><code>$(function(){
      var i, len, arr, temp, m_board;
  
      m_board=IsepicChess.initBoard({name : "main"});
  
      arr=["e4-e2", "e7-e5", "f3-g1"]; //is ok for the initial and final squares to be swapped
  
      for(i=0, len=arr.length; i&lt;len; i++){
          temp=arr[i].split("-");
  
          //one of the two following lines will silently fail
          m_board.moveCaller(temp[0], temp[1]);
          m_board.moveCaller(temp[1], temp[0]);
      }
  
      m_board.refreshBoard();
  });
  </code></pre>
  
  <p>.4. copy the move list and add some PGN headers above it like <code>[WhiteElo "2400"]</code> etc.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2014-03-18T21:18:04.940Z
Answers.LastActivityDate: 2019-10-17T04:26:18.280Z
Answers.LastEditDate: 2019-10-17T04:26:18.280Z
Answers.UserDisplayName: ajax333221
Answers.Body: <p>Joey Meyer wrote and built &quot;Raspberry Turk&quot;:</p>
  <blockquote>
  <p>The Raspberry Turk is a robot that can play chess—it's entirely open
  source, based on Raspberry Pi, and inspired by the 18th century chess
  playing machine, the Mechanical Turk.</p>
  </blockquote>
  <p>This goes beyond the OP's question because it also incorporates (a) the engine to calculate responses and (b) a robotic arm to make the engine's moves on the board.</p>
  <p>But it answers the OP's question because one component of Raspberry Turk includes an overhead camera and a machine-learning component to learn how to convert what it sees on the board into a PGN representation of the board.</p>
  <ul>
  <li>You can see it in action at this YouTube video <a href="https://www.youtube.com/watch?v=lpJIBVU_WJE" rel="nofollow noreferrer">&quot;Chess Playing Robot Powered by Raspberry Pi - Raspberry Turk&quot;</a>. (Note that the video is sped up by about a factor of 12, per personal communication with the creator.)</li>
  <li>For an accessible explanation, see Chaim Gartenberg, &quot;<a href="https://www.theverge.com/circuitbreaker/2017/3/24/15049154/raspberry-turk-diy-chess-robot-learn-coding-machine-learning-python-robotics" rel="nofollow noreferrer">The Raspberry Turk is a DIY chess robot with instructions to build your own</a>*,&quot; The Verge, March 24, 2017.</li>
  <li>The code is available at <a href="https://github.com/joeymeyer/raspberryturk" rel="nofollow noreferrer">GitHub » joeymeyer/raspberryturk</a></li>
  <li>The project website is <a href="http://www.raspberryturk.com/" rel="nofollow noreferrer">raspberryturk.com/</a></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-15T01:18:31.677Z
Answers.LastActivityDate: 2020-10-15T01:18:31.677Z
Answers.UserDisplayName: Jim Ratliff
Answers.Body: <p>I'm the developer of ChessCam, an open-source web app to digitize your chess games. It relies mainly on running object detection models (yolov8) in the browser.</p>
  <p>Try it out here: <a href="https://www.chesscam.net" rel="nofollow noreferrer">https://www.chesscam.net</a> (code at <a href="https://github.com/Pbatch/CameraChessWeb" rel="nofollow noreferrer">https://github.com/Pbatch/CameraChessWeb</a>).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-15T23:34:38.547Z
Answers.LastActivityDate: 2023-12-15T23:34:38.547Z
Answers.UserDisplayName: PBatch
Answers.Body: <p>Firstly, great post. I have been surveying this issue recently and the answers here are a great source of information.
  I have found an excellent free app for Android and iOS called KnightVision. It is relatively new. It was trained on various chess sets. It even managed to properly recognize my hand-crafted pieces which are not identical to the Staunton standard. It works well, although to be on the safe side, it requires a separation of about 3 seconds between each move on the board. So be gentle when swapping pieces, which is something that is usually done quite quickly.
  It also has a web page, where after logging in you can see the game PGN that you can copy and export to any chess engine for studying. It also has some analysis on the website itself.
  <a href="https://www.knightvision.io/" rel="nofollow noreferrer">https://www.knightvision.io/</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-16T09:31:35.453Z
Answers.LastActivityDate: 2022-11-17T11:03:31.593Z
Answers.LastEditDate: 2022-11-17T11:03:31.593Z
Answers.UserDisplayName: Ori Honigman
Answers.Body: <p><em>I won't be answering the question, because I don't know if there's a software out there. I'm just saying how I think that's how to do it, I don't know for sure, so...</em></p>
  
  <hr>
  
  <p>Okay first you need to make sure that the video is recorded from a chess software, not a live tournament, just to make your job easier.</p>
  
  <p>You need to convert the movie to images</p>
  
  <p><code>ffmpeg -i foo.avi -vsync 0 -vf select="eq(pict_type\,PICT_TYPE_I)" -s WxH -f image2 foo-%03d.jpeg</code></p>
  
  <ul>
  <li><a href="https://stackoverflow.com/questions/12100072/how-to-extract-slides-from-a-video-using-python">Source</a></li>
  </ul>
  
  <p>That was easy, now the hard part, you need to start doing image processioning, for that, the most powerful is OpenCV.</p>
  
  <p>For that, there are several techniques, I'll give you the links so you can read them.</p>
  
  <ul>
  <li><a href="http://codebazaar.blogspot.com/2011/08/chess-board-recognition-project-part-1.html" rel="nofollow noreferrer">Chessboard picture recognition project</a></li>
  <li><a href="https://stackoverflow.com/questions/11288460/recommended-pattern-recognition-technique-for-chess-board">Recommended pattern recognition technique for chess board</a></li>
  <li><a href="https://dsp.stackexchange.com/questions/2805/how-does-opencv-find-chessboard-corners">How does OpenCV find chessboard corners?</a></li>
  <li><a href="http://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=1&amp;cad=rja&amp;ved=0CCgQFjAA&amp;url=http://www.researchgate.net/publication/234491319_Using_Image_Processing_Techniques_to_Automate_Chess_Game_Recording/file/79e4150fe3be5bef54.pdf&amp;ei=s1YCU8KLL-jI0AWQ2YDQCw&amp;usg=AFQjCNHTgvncb4RBq830y9dgXJiN7uCYtg&amp;sig2=2duB1naDe__3OaMIrV40pA" rel="nofollow noreferrer">Using Image Processing Techniques to Automate Chess Game
  Recording</a></li>
  <li><a href="http://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=15&amp;cad=rja&amp;ved=0CIEBEBYwDg&amp;url=http://www.mdpi.com/1424-8220/10/3/2027/pdf&amp;ei=s1YCU8KLL-jI0AWQ2YDQCw&amp;usg=AFQjCNFYCCakOaa5v52uV7pedsdhAGJpKQ&amp;sig2=syKHZoJRHHQqHPeh4Na4-Q" rel="nofollow noreferrer">Automatic Chessboard Detection for Intrinsic and Extrinsic Camera
  Parameter Calibration</a></li>
  <li><a href="https://opencv-python-tutroals.readthedocs.org/en/latest/py_tutorials/py_calib3d/py_calibration/py_calibration.html" rel="nofollow noreferrer">Camera Calibration and 3D Reconstruction</a></li>
  </ul>
  
  <p>It is complicated but simpler than you think, because you only need to know what is the starting position, loop through all the images and see the difference. Take a look at OpenCV docs you'll find tons of materials on that subject.</p>
  
  <hr>
  
  <p><em>Oh by the way, what you're asking for is done already, but not using software as far as I know, there's an old project that was coded using Pascal, cameras placed over the board, robots do image processing and make a move. Nowadays many robots play chess.</em></p>
  
  <p><em>You could buy an Arduino uno,  and its light censor, it will detect the white pieces and the black pieces, some coding and it will play chess. :)</em></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2014-02-17T19:05:11.923Z
Answers.LastActivityDate: 2014-02-17T19:17:05.813Z
Answers.LastEditDate: 2017-05-23T12:37:05.403Z
Answers.UserDisplayName: Lynob
Answers.Body: <p><a href="https://www.kickstarter.com/projects/infivention/square-off-neo-and-swap-board-games-powered-by-robotics-and-ai?ref=12ukse&amp;utm_source=KickstarterUpdate&amp;utm_medium=UpdateEmail&amp;utm_campaign=WeAreBack" rel="nofollow noreferrer">Squareoff</a> is making another board at 200$. Expensive but not extortionist. They say it will be available March 2021. Automatically moves pieces too. <a href="https://squareoffnow.com/product/neo" rel="nofollow noreferrer">Details.</a> Has Blitz mode but may not be so good for it. May not allow more than one user per board but plays through chess.com and maybe lichess.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-14T05:21:48.753Z
Answers.LastActivityDate: 2020-10-16T01:52:09.610Z
Answers.LastEditDate: 2020-10-16T01:52:09.610Z
Answers.UserDisplayName: ran8
Answers.Body: <p>I use the <a href="http://monroi.com/products/personal-chess-manager.html" rel="nofollow">Monroi Chess Notation Device</a> for all my chess games and it is basically an electronic board.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2014-02-22T11:18:40.060Z
Answers.Text: That doesn't answer the OP's question, though.
Answers.UserDisplayName: Dag Oskar Madsen
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2014-02-23T02:30:44.387Z
Answers.Text: you could use a digital presenter to record a game its a camera just above the table. then you could just write an application tracking the objects in a grid.
Answers.UserDisplayName: Cferrel
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2014-02-22T05:45:22.117Z
Answers.LastActivityDate: 2014-03-01T19:42:32.400Z
Answers.LastEditDate: 2014-03-01T19:42:32.400Z
Answers.UserDisplayName: Cferrel
Body: <p>Sometimes during blitz games over the board, or when in zeitnot, it gets very difficult to annotate them.</p>
  
  <p>I've seen several software / hardware to help in writing the notation, even <a href="http://monroi.com/products/personal-chess-manager.html" rel="noreferrer">one that's FIDE approved</a>; these require that you repeat the moves on the device. Hardly faster than pen and paper, just as prone to errors, plus you risk typing the wrong move and wasting even more time.  </p>
  
  <p><a href="http://www.chess.it/soft/dgt.htm" rel="noreferrer">DGT makes a board</a> that will do this automatically, but it will only work with its pieces, and starts at 400€, double that for <a href="http://www.dgt.nl/site/index.php/products/electronic-boards/serial-tournament" rel="noreferrer">tournament boards</a>.  Some independent projects are available such as <a href="http://www.chess.com/forum/view/chess-equipment/usb-chess-board-101" rel="noreferrer">usb chessboard 101</a>, or <a href="http://hackaday.com/2013/05/09/voice-controlled-chess-robot/" rel="noreferrer">voice controlled chess robots</a>, <a href="http://hackaday.com/2012/04/18/chess-board-step-sequencer/" rel="noreferrer">Chess board step sequencer</a>.  Additionally some <a href="http://hackadaycom.files.wordpress.com/2011/05/final_project.pdf" rel="noreferrer">open projects are available</a> that even move pieces (or <a href="http://en.chessbase.com/post/you-built-what-a-giant-robotic-chess-set" rel="noreferrer">gigantic pieces</a>).</p>
  
  <p>I have a playoff tonight, and the tournament rules dictate shorter and shorter games until there is a winner.  Hence I thought about shooting a movie of the board should we lose track.  But then my programming background kicked in and I thought "there must be some software that can process a movie and write the PGN". </p>
  
  <p>I searched everywhere, and found nothing. </p>
  
  <ol>
  <li>Does any such thing exist (webcam/video to pgn converter)? A prototype maybe? </li>
  <li>Has anyone attempted such a feat and shared their research?</li>
  <li>Are there obvious limitations that make this project unlikely to succeed?</li>
  <li>Would you buy it if it existed?</li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2015-02-26T11:13:58.310Z
Comments.Text: actually it's simply a linear transformation to "unstretch" the board and make all its squares square. the main problem is to recognize the square the piece is moved from if the angle is such that pieces can line up behind each other.
Comments.UserDisplayName: peter
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-21T05:03:14.080Z
Comments.Text: Given that https://www.kickstarter.com/projects/infivention/square-off-worlds-smartest-chess-board-relaunched was already successful a crowdfunding project seems feasible indeed. Especially since the cost for end users would be very low.
Comments.UserDisplayName: Wolfgang Fahl
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-18T03:33:43.767Z
Comments.Text: The wording of the question right now is 4 different questions at once. Is there some way that you could consolidate them as one, or perhaps pick one in particular and use the others as points to consider?
Comments.UserDisplayName: Charles Rockafellor
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2014-02-11T12:10:17.927Z
Comments.Text: it would be an interesting computer vision / machine learning project!
Comments.UserDisplayName: Memming
Tags: <software><tournament><notation><pgn>
Title: Tools for automated notation: webcam movie to PGN
ThreadText: # Tools for automated notation: webcam movie to PGN
  
  (Asked by: username_0 on 2014-02-09)
  
  Sometimes during blitz games over the board, or when in zeitnot, it gets very difficult to annotate them.
  
  I've seen several software / hardware to help in writing the notation, even [one that's FIDE approved](http://monroi.com/products/personal-chess-manager.html); these require that you repeat the moves on the device. Hardly faster than pen and paper, just as prone to errors, plus you risk typing the wrong move and wasting even more time.
  
  [DGT makes a board](http://www.chess.it/soft/dgt.htm) that will do this automatically, but it will only work with its pieces, and starts at 400€, double that for [tournament boards](http://www.dgt.nl/site/index.php/products/electronic-boards/serial-tournament). Some independent projects are available such as [usb chessboard 101](http://www.chess.com/forum/view/chess-equipment/usb-chess-board-101), or [voice controlled chess robots](http://hackaday.com/2013/05/09/voice-controlled-chess-robot/), [Chess board step sequencer](http://hackaday.com/2012/04/18/chess-board-step-sequencer/). Additionally some [open projects are available](http://hackadaycom.files.wordpress.com/2011/05/final_project.pdf) that even move pieces (or [gigantic pieces](http://en.chessbase.com/post/you-built-what-a-giant-robotic-chess-set)).
  
  I have a playoff tonight, and the tournament rules dictate shorter and shorter games until there is a winner. Hence I thought about shooting a movie of the board should we lose track. But then my programming background kicked in and I thought "there must be some software that can process a movie and write the PGN".
  
  I searched everywhere, and found nothing.
  
  1. Does any such thing exist (webcam/video to pgn converter)? A prototype maybe?
  2. Has anyone attempted such a feat and shared their research?
  3. Are there obvious limitations that make this project unlikely to succeed?
  4. Would you buy it if it existed?
  
  **Tags:** software, tournament, notation, pgn
  
  > **Comments:**
  > **username_4**: it would be an interesting computer vision / machine learning project!  
  > **username_1**: actually it's simply a linear transformation to "unstretch" the board and make all its squares square. the main problem is to recognize the square the piece is moved from if the angle is such that pieces can line up behind each other.  
  > **username_2**: Given that https://www.kickstarter.com/projects/infivention/square-off-worlds-smartest-chess-board-relaunched was already successful a crowdfunding project seems feasible indeed. Especially since the cost for end users would be very low.  
  > **username_3**: The wording of the question right now is 4 different questions at once. Is there some way that you could consolidate them as one, or perhaps pick one in particular and use the others as points to consider?  
  
  ---
  
  ## 12 answers
  
  ### Answer by username_2 on 2019-10-15. Score: 7
  
  It's 2019-12 now and 5 years after the original question.
  
  1. Does any such thing exist (webcam/video to pgn converter)? A
     prototype maybe?
  
     see
  
     * <https://github.com/WolfgangFahl/play-chess-with-a-webcam/issues/19>
     * <https://github.com/WolfgangFahl/play-chess-with-a-webcam/issues/22>
       which are both closed by now. See <http://lichess.bitplan.com> for a game successfully recorded. Join <https://gitter.im/play-chess-with-a-webcam/community> to discuss
  2. Has anyone attempted such a feat and shared their research?
  
     Yes - so far I found some 35 research papers and some 10 open source solutions and one commercial offering
  
     2.1 <http://wiki.bitplan.com/index.php/PlayChessWithAWebCam/Papers>
  
     2.2. <http://wiki.bitplan.com/index.php/PlayChessWithAWebCam#Open_Source_Projects>
  
     2.3. <http://wiki.bitplan.com/index.php/PlayChessWithAWebCam#Commercial>
  
     <http://wiki.bitplan.com/index.php/PlayChessWithAWebCam> is another project
     aiming at the goals stated by you. The project was started 2019-10-15 and in a few hours I already found a lot of resources e.g. this Question
     Some two months later I have updated my answer (see links above)
  3. Are there obvious limitations that make this project unlikely to succeed?
     I can only answer from my own experience with a similar project I did 31 years ago see <http://wiki.bitplan.com/index.php/MagneticSensorChessBoard1987>.
     At that time i tried to create a chess board that would detect moves. I used magnets
     in the wooden pieces and reed contacts. The key to success was using a chess engine
     that new "the rules" to be able to concentrate on movements alone and not having
     to detect individual pieces. As outlined in an answer above promotions to non queen
     pieces might be one of the edge usecases that are difficult. Detecting the motion of
     pieces via picture differences is a well understood problem.
  
     Chess engines and UCI compatible GUIs are also available these days. So the problem
     is more an integration problem these days then solving the individual detail issues.
  
     A major obstacle is the difference between a "prototype" and a product. See e.g.
     <http://www.nandanbanerjee.com/files/ICCVR-08AUG12-011%20paper.pdf>
  
     a paper that is even older than this question but far from a product that you can buy
     at a reasonable price.
  4. Would you buy it if it existed?
  
     Yes. When I bought a new chess board a few days ago I was looking for one that
     could record moves and it was still around 500 EUR. As it looks the hardware cost
     will be well below 150 EUR these days including a decent chess board and pieces.
  
     The software might well be opensource and free (as in free beer) soon. I'll update
     this answer accordingly.
  
  > **Comments:**
  > **username_0**: Thank you, I'm looking forward to seeing your progress  
  > **username_5**: I suggest to add to the knowledge, the following website:   https://github.com/maciejczyzewski/neural-chessboard  
  > **username_2**: @username_5 - thanks for the hint i think it is already part of http://wiki.bitplan.com/index.php/PlayChessWithAWebCam/Papers  
  
  ---
  
  ### Answer by username_6 on 2014-02-10. Score: 5
  
  In order,
  
  1) Not to my knowledge
  
  2) Not to my knowledge
  
  3) There are several things that make this sort of project non-trivial, even difficult. Not the least of which is following the piece, differentiating it from a hand gesture, for example. Also determining when the move stopped would be an interesting question for computer vision to solve. These kinds of problems have been solved in other situations, but not with 100% accuracy, and certainly not at prices that are even close to being competitive with the DGT board.
  
  Then there's the question of where the camera could be positioned. Directly overhead would offer the best view of the board, making it easier to determine the square with precision. But wherever the camera gets placed, there's a distinct probability its view will be obstructed by one of the players, either leaning over the board, reaching for the clock, or simply gesticulating.
  
  4) Not unless it was really cheap. And I don't expect it would be. My reasoning is simple. As time controls shorten, the quality of the game lessens, to the point where it's no longer interesting to me
  
  > **Comments:**
  > **username_0**: You make a great point at 4: the quality is indeed poor.  Not worth the hassle of setting up a camera, much less spending a month to write the software.  I just won the playoff, but I am embarassed of how poorly I played.  
  
  ---
  
  ### Answer by username_7 on 2015-06-15. Score: 4
  
  I wrote this software. It contains webcam AI for move recognition, connected to Arena GUI.
  
  <https://www.youtube.com/watch?v=APkoE1UDKj0>
  
  > **Comments:**
  > **username_0**: Very impressive username_7, congratulations! Is the alpha available for testing already? Can you expose the moves through a service, for use by other software?  
  
  ---
  
  ### Answer by username_8 on 2014-02-09. Score: 4
  
  That would be a rather complex task. Maybe a reasonable first step is to look into recognizing FEN position from a single image - for which more resources exists, for example:
  
  * [Create a FEN from a chess diagram picture](http://chessgrabber.nicolaas.net/acknowledgements)
  * [Recommended pattern recognition technique for chess board](https://stackoverflow.com/questions/11288460/recommended-pattern-recognition-technique-for-chess-board)
  
  > **Comments:**
  > **username_0**: Excellent thank you. I read both and they are really interesting.  Chessgrabber is a recent project and is definitely more suited, I contacted the developer.  
  > **username_9**: I'd think that from this first step it wouldn't be too difficult to convert a video to PGN - take images every few milliseconds, analyse them, and if the position has changed in a legal way add the new position to the file.  
  > **username_0**: Update: the developer responded, it will never work for 3-d skewed images with overlapping pieces. Too bad, without that as a base, the project will never start.  
  > **username_1**: recognizing a still image is actually harder than having the video leading up to it, since the latter eliminates the problem of having to be able to identify the pieces.  
  
  ---
  
  ### Answer by username_10 on 2016-09-09. Score: 3
  
  I worked on the chess tracking project **[8x8cam](http://iie.fing.edu.uy/investigacion/grupos/gti/timag/trabajos/2013/ajedrez/index.html)** within image processing course in 2013 at FING UdelaR.
  
  Then in 2014 other students created a better version for Android named **[chesstrack](http://iie.fing.edu.uy/investigacion/grupos/gti/timag/trabajos/2014/ajedrez/index.html)**. Under it's web you can find videos and link to the code they used.
  
  I would love to see a finalized app of this project idea I had for years. So if there is some news about this please leave a comment.
  
  Answering your pointed questions:
  
  1. Does any such thing exist (webcam/video to pgn converter? A prototype maybe?
  
     Apart from my project [8x8cam](http://iie.fing.edu.uy/investigacion/grupos/gti/timag/trabajos/2013/ajedrez/index.html) and [chesstrack](http://iie.fing.edu.uy/investigacion/grupos/gti/timag/trabajos/2014/ajedrez/index.html), I have recently found a project named [ChessCam](http://blogdugas.net/blog/2015/05/18/play-chess-with-a-webcam/). All of these projects are Open Source and only need to be named as resource if you create a new project from them.
  2. Has anyone attempted such a feat and shared their research?
  
     Yes. You can see such on previous question's answer.
  3. Are there obvious limitations that make this project unlikely to succeed?
  
     The hardest part I believe is to make the software work no matter how the camera is placed and independently of the illumination. And also to work on fast paced games and work with the hand occlusions when the game is being played, in order to detect correctly the move done.
  4. Would you buy it if it existed?
  
     Definitely yes.
  
  
  ---
  
  ### Answer by username_11 on 2014-03-18. Score: 1
  
  Edit: promoting to anything different than queen will make everything explode. It will also be very hard to support this (my program for Part 2 supports underpromotion, but storing the info of Part 1 purely by reading pixels is a nightmare).
  Edit2: I just noticed you mean real board chess to PGN, so this is all useless and I will most likely delete my answer.
  Part 1: storing the squares using pixel readers
  It is totally possible, the steps needed would be:
  
  change the set up of the board so it highlights the last move (prefereably not just the border)
  write a program that reads pixels on given (x,y)
  make it read about 5+ pixels on each square (so it is impossible for the cursor/piece to cover all the pixels to be read)
  convert the HEX to RBG, then give a range of accepted values of what we deem to be considered close to the "last move" color
  save the moves on an array with a format ["h1-h8", "e4-e5", ...]  just the initial and ending squares (order unimportant)
  
  Note: from here we can't really know from which square was the initial/ending position, but there are only 2 posibilities and only 1 of them is legal.
  I have written simple pixel-based macros/bots for RPGs games in C#, I can guarantee this is not that difficult, given that:
  
  the cam is static (so the x,y values don't get messed up)
  you previously know the background color or you can set it up before playing
  
  Basically, it is best if the game is played online and you are recording directly from the screen.
  
  Sorry, no program avaiable for part #1
  
  
  Part 2: exporting that list into PGN
  To turn this list into a real PGN, you can use a program I wrote in JavaScript that records moves, and one of the functions moveCaller() you pass two squares positions and it searches the final_pos in the legal moves of initial_pos and if there is a match then it calls makeMove(). So we can call it twice e5 to e4 and e4 to e5 and I guarantee it will ignore the illegal one.
  
  download the project (direct link .zip)
  uncompress the .zip
  change the JavaScript of the index.html file with this:
  
  '
  $(function(){
      var i, len, arr, temp, m_board;
  
      m_board=IsepicChess.initBoard({name : "main"});
  
      arr=["e4-e2", "e7-e5", "f3-g1"]; //is ok for the initial and final squares to be swapped
  
      for(i=0, len=arr.length; i
  .4. copy the move list and add some PGN headers above it like [WhiteElo "2400"] etc.
  
  
  ---
  
  ### Answer by username_12 on 2020-10-15. Score: 1
  
  Joey Meyer wrote and built "Raspberry Turk":
  
  > The Raspberry Turk is a robot that can play chess—it's entirely open
  > source, based on Raspberry Pi, and inspired by the 18th century chess
  > playing machine, the Mechanical Turk.
  
  This goes beyond the OP's question because it also incorporates (a) the engine to calculate responses and (b) a robotic arm to make the engine's moves on the board.
  
  But it answers the OP's question because one component of Raspberry Turk includes an overhead camera and a machine-learning component to learn how to convert what it sees on the board into a PGN representation of the board.
  
  * You can see it in action at this YouTube video ["Chess Playing Robot Powered by Raspberry Pi - Raspberry Turk"](https://www.youtube.com/watch?v=lpJIBVU_WJE). (Note that the video is sped up by about a factor of 12, per personal communication with the creator.)
  * For an accessible explanation, see Chaim Gartenberg, "[The Raspberry Turk is a DIY chess robot with instructions to build your own](https://www.theverge.com/circuitbreaker/2017/3/24/15049154/raspberry-turk-diy-chess-robot-learn-coding-machine-learning-python-robotics)\*," The Verge, March 24, 2017.
  * The code is available at [GitHub » joeymeyer/raspberryturk](https://github.com/joeymeyer/raspberryturk)
  * The project website is [raspberryturk.com/](http://www.raspberryturk.com/)
  
  
  ---
  
  ### Answer by username_13 on 2023-12-15. Score: 1
  
  I'm the developer of ChessCam, an open-source web app to digitize your chess games. It relies mainly on running object detection models (yolov8) in the browser.
  
  Try it out here: <https://www.chesscam.net> (code at <https://github.com/Pbatch/CameraChessWeb>).
  
  
  ---
  
  ### Answer by username_14 on 2022-11-16. Score: 1
  
  Firstly, great post. I have been surveying this issue recently and the answers here are a great source of information.
  I have found an excellent free app for Android and iOS called KnightVision. It is relatively new. It was trained on various chess sets. It even managed to properly recognize my hand-crafted pieces which are not identical to the Staunton standard. It works well, although to be on the safe side, it requires a separation of about 3 seconds between each move on the board. So be gentle when swapping pieces, which is something that is usually done quite quickly.
  It also has a web page, where after logging in you can see the game PGN that you can copy and export to any chess engine for studying. It also has some analysis on the website itself.
  <https://www.knightvision.io/>
  
  
  ---
  
  ### Answer by username_15 on 2014-02-17. Score: 0
  
  *I won't be answering the question, because I don't know if there's a software out there. I'm just saying how I think that's how to do it, I don't know for sure, so...*
  
  ---
  
  Okay first you need to make sure that the video is recorded from a chess software, not a live tournament, just to make your job easier.
  
  You need to convert the movie to images
  
  `ffmpeg -i foo.avi -vsync 0 -vf select="eq(pict_type\,PICT_TYPE_I)" -s WxH -f image2 foo-%03d.jpeg`
  
  * [Source](https://stackoverflow.com/questions/12100072/how-to-extract-slides-from-a-video-using-python)
  
  That was easy, now the hard part, you need to start doing image processioning, for that, the most powerful is OpenCV.
  
  For that, there are several techniques, I'll give you the links so you can read them.
  
  * [Chessboard picture recognition project](http://codebazaar.blogspot.com/2011/08/chess-board-recognition-project-part-1.html)
  * [Recommended pattern recognition technique for chess board](https://stackoverflow.com/questions/11288460/recommended-pattern-recognition-technique-for-chess-board)
  * [How does OpenCV find chessboard corners?](https://dsp.stackexchange.com/questions/2805/how-does-opencv-find-chessboard-corners)
  * [Using Image Processing Techniques to Automate Chess Game
    Recording](http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&ved=0CCgQFjAA&url=http://www.researchgate.net/publication/234491319_Using_Image_Processing_Techniques_to_Automate_Chess_Game_Recording/file/79e4150fe3be5bef54.pdf&ei=s1YCU8KLL-jI0AWQ2YDQCw&usg=AFQjCNHTgvncb4RBq830y9dgXJiN7uCYtg&sig2=2duB1naDe__3OaMIrV40pA)
  * [Automatic Chessboard Detection for Intrinsic and Extrinsic Camera
    Parameter Calibration](http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=15&cad=rja&ved=0CIEBEBYwDg&url=http://www.mdpi.com/1424-8220/10/3/2027/pdf&ei=s1YCU8KLL-jI0AWQ2YDQCw&usg=AFQjCNFYCCakOaa5v52uV7pedsdhAGJpKQ&sig2=syKHZoJRHHQqHPeh4Na4-Q)
  * [Camera Calibration and 3D Reconstruction](https://opencv-python-tutroals.readthedocs.org/en/latest/py_tutorials/py_calib3d/py_calibration/py_calibration.html)
  
  It is complicated but simpler than you think, because you only need to know what is the starting position, loop through all the images and see the difference. Take a look at OpenCV docs you'll find tons of materials on that subject.
  
  ---
  
  *Oh by the way, what you're asking for is done already, but not using software as far as I know, there's an old project that was coded using Pascal, cameras placed over the board, robots do image processing and make a move. Nowadays many robots play chess.*
  
  *You could buy an Arduino uno, and its light censor, it will detect the white pieces and the black pieces, some coding and it will play chess. :)*
  
  
  ---
  
  ### Answer by username_16 on 2020-10-14. Score: 0
  
  [Squareoff](https://www.kickstarter.com/projects/infivention/square-off-neo-and-swap-board-games-powered-by-robotics-and-ai?ref=12ukse&utm_source=KickstarterUpdate&utm_medium=UpdateEmail&utm_campaign=WeAreBack) is making another board at 200$. Expensive but not extortionist. They say it will be available March 2021. Automatically moves pieces too. [Details.](https://squareoffnow.com/product/neo) Has Blitz mode but may not be so good for it. May not allow more than one user per board but plays through chess.com and maybe lichess.
  
  
  ---
  
  ### Answer by username_17 on 2014-02-22. Score: -1
  
  I use the [Monroi Chess Notation Device](http://monroi.com/products/personal-chess-manager.html) for all my chess games and it is basically an electronic board.
  
  > **Comments:**
  > **username_18**: That doesn't answer the OP's question, though.  
  > **username_17**: you could use a digital presenter to record a game its a camera just above the table. then you could just write an application tracking the objects in a grid.  
  
  ---
  
  