Thanks for sharing your enthusiasm and contributing to the TensorFlow.js ecosystem! This sounds like a really interesting project. Just a heads up a lot of our engineers are out of office for the holidays this week so it may be best to pick this up in the new year.
On a somewhat related note however (I know this is not handwriting recognition) I have seen a promising library for OCR come out in TFJS recently from a group of folk over in France. They may have some pointers for guidance for you as similar domain in the meantime.

  
      

      GitHub
  

  
    

GitHub - mindee/doctr-tfjs-demo: Javascript demo of docTR, powered by... 1

  Javascript demo of docTR, powered by TensorFlowJS. Contribute to mindee/doctr-tfjs-demo development by creating an account on GitHub.


  

  
    
    
  

  


PS you said you had a demo somewhere - is it somewhere I can try for handwriting / speech recognition?