I was talking to one of our SWEs on the team and they suggested for you to try converting your layers model to a graph model first with the TensorFlow.js converter.
Check this documentation here:

  
      

      github.com
  

  
    tfjs-converter/tfjs-converter at master · tensorflow/tfjs-converter 1

  master/tfjs-converter

  Convert TensorFlow SavedModel and Keras models to TensorFlow.js

  

  
    
    
  

  


In this situation you could use --input_format keras and then set the --output_format tfjs_graph_model  as frozen model should be suitable for inference in browser as you do not need to retain in the browser.
Let me know if that helps!