TFLite is not meant to perform good on the commodity hardware actually. It’s opset is optimized to run faster primarily on mobile hardware. However, if you build TFLite for your platform (preferably with XNNPACK enabled) then you may get some benefits.
Here’s some more information:

  
      

      TensorFlow
  

  
    

Build TensorFlow Lite with CMake 7