I’m looking to potentially transition from TF2.0 to using pytorch for a variety of reasons but before I do (and before I pitch this idea to the group) I need to sort out some potential sticking points, the main one being the production side.
I would like to do all of my development and testing in python then deploy to use in C++. The use case is as follows
Windows standalone application running 1 or more models in inference in C++ built using MSVC. Data is opencv Mats for input to the model and output. Models are usually segmentation models (so same shape in and out) or object detection models (or instance models likes Mask-RCNN).
What would be the best path with this currently. I see alot of ways forward with libtorch, torchscript, ONNX to Azure stuff, ONNX to Caffe2. I have gotten confused at this point.
I want the simplest way with good Windows support. Tensorflow has been a headache with a lack of C++ documentation and little to no windows support (breaks constantly with releases).
I also want something that’s fast and scalable. In the future I might be running distributed, cloud, mixed precision, TensorRT etc.
It’s a fairly small codebase and team so API stability isn’t super critical.