Great summary! Thanks for your interest in PyTorch 
To give some context to this bullet:


TorchScript works well for simple architectures, but there is work needed to support more complex models like objection detectors and LSTMs


We have confidence that TorchScript is expressive enough to support complex models, but you’re correct that some legwork is involved. We showcased some work with LSTMs on our blog 15, and in the next version of Torchvision I believe all the models (including object detection models like MaskRCNN) will be TorchScript compatible.
As for your use case specifically, we have reason to believe PyTorch is a good fit; Uber ATG’s perception and prediction network uses a pure-TorchScript implementation, which (for obvious reasons ) has very similar requirements to yours. @sidazhang gave an amazing talk 39 at our devcon about their experience using PyTorch for their end-to-end research to production.
The steps you outlined for moving forward seem like the right ones overall. I would add only that in between “check libtorch performance” and “try TensorRT if it’s not fast enough”, there are a number of ways to tune your PyTorch model for performance. Would be happy to discuss further when you get there!