libtorch is the C++ frontend while custom C++/CUDA extensions can be used in Python as modules or functional ops. Yes, both can be used for inference.


@tom’s general rule is: “As long as your operands are reasonably large (say 100s of elements, not single elements), Python and Data “administrative overhead” probably isn’t your main problem.”
This would mean that unless your actual workload is small you shouldn’t see a huge difference between using the Python vs. C++ interface, but tom can correct me here. 


You could either script the model or write it directly in libtorch. As a quick test: try to export the model via torch.jit.script(model) and see if you are running into issues.


libtorch and custom C++ extensions have a different purpose, so I don’t think it’s a choice between these two.