Well my main point is that you may be able to solve your problem using regular computer vision techniques depending how clean your image data is etc. If you find that is not working well, then I would try solving with Machine Learning using TensorFlow.js but also ensure the machine learning solution beats your “baseline” of the non - ML solution if that makes sense?
What you want to do is certainly achievable with TensorFlow.js though, and in real time too. Check out this interview I did with one member of the community who explains really well how he made a Soduku solver that is actually very similar problem as yours:
            

If you check the description links of the video too there is one more video where he goes even deeper about the preprocessing steps he takes (non ML tasks) along with his TensorFlow.JS implementation that will help you too:
Browser-Based Augmented Reality Sudoku Solver using TensorFlow and Image Processing