Thank you so much for your reply! I’m continually surprised and delighted by the responsiveness of the PyTorch team and community, even as the project matures! It’s a great time to be a PyTorch user .
So if I were to summarize, the future for PyTorch deployment would look like this:

Users prototype models in Python, annotate them with TorchScript, and get a high-performance C++ runtime out of the box thanks to JIT
As JIT matures, PyTorch developers will increasingly focus on performance and try to close the gap with existing backends (TensorRT, TVM, Glow)
JIT performance will ideally be so good, users won’t have to integrate specialized backends, except in rare cases where they need that last drop of speed
If users DO need to integrate with other backends, PyTorch will make it easy to incorporate them, enabling the use of different compilers for different submodules in a graph (e.g. pytorch/tvm repo)
PyTorch will support ONNX for interoperability, but ideally JIT will make this unnecessary for use cases where performance is the goal

Does that look right? And to summarize the current state as of October 2019:

JIT performance is sufficient for many use cases, but not yet on par with other compilers (TVM, Glow, TensorRT)
TorchScript works well for simple architectures, but there is work needed to support more complex models like objection detectors and LSTMs
For those of us training object detectors, we have 2 options:
– Change our python code to be compatible with the ops currently supported by TorchScript
– Implement a custom TorchScript operation (docs 11)
If JIT performance isn’t sufficient and we need to squeeze the last drop of performance, the recommended approach is to add a custom compiler yourself (e.g. pytorch/tvm 19)
Another way to increase performance, for situations where you have a stable backbone, is to first compile the backbone with TVM/TensorRT, and then write a custom TorchScript op in your model (e.g. https://tutorials.pytorch.kr/advanced/torch_script_custom_ops.html 11 <-- I think?)
A final way to increase performance (but not recommended), is to convert to ONNX, and then compile with a supported backend from there

And just to summarize my use case and try to answer my own question:
Use case

Rapidly prototype SOTA object detectors in Python
Deploy to a C++ environment with strict latency requirements

Recommendation

Update my python code to be compatible with TorchScript (example here 14) or write custom TorchScript ops
Add libtorch as a dependency in my stack and benchmark performance against current models
If performance is not sufficient, try porting the backbone to TensorRT and add a custom op in TorchScript
If performance is still not sufficient, write a custom compiler for TensorRT/TVM to compile specific ops (e.g. pytorch/tvm 19)
If all else fails, try converting to ONNX and explore converting to TensorRT/TVM from there (or try ONNXruntime)

Thanks again! Let me know if I’m missing anything!