I think there is plenty of room to improve on the standard “use flask” advice and the solutions they present. Without leaving Python, one can do many things right (like request batching, using a framework with async etc.). Christian also has a great slide deck that lists some desiderata 24.
I think it is hard to completely avoid copying if you want your server to decode images (because you have request -> JPEG format blob -> Tensor -> Rescaled), but many of the other items from Christian’s list can be handled quite nicely using Python + traced models (and gRPC could probably be used to eliminate some of the copying headaches). The JIT liberates you from annoyances with the GIL during the running of your model.
Best regards
Thomas