Setting up a specialised server just for predictions feels like an overkill.

I don’t think that it is overkill and probably the alternatives are not so simpler than Tensorflow Serving with Docker
E.g. see:
  
      

      Medium – 8 Feb 21
  

  
    

ML model deployment option with concurrency (Flask + uWSGI) 1

  Beginner’s guide to ML solutions deployment maintaining concurrency + load testing. Discussion on the Data Scientist’s responsibilities…

  
    Reading time: 9 min read